sysadmin/openfreemap
1
0
Fork 0
mirror of https://github.com/hyperknot/openfreemap.git synced 2026-05-21 14:02:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
Zsolt Ero
2024-08-27 02:02:37 +02:00
parent 08d17df476
commit b746263cea
4 changed files with 13 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@@ -56,7 +56,7 @@ There is no cloud, just dedicated servers. The HTTPS server is nginx on Ubuntu.
Production-quality hosting of 300 million tiny files is hard. The average file size is just 450 byte. Dozens of tile servers have been written to tackle this problem, but they all have their limitations. Production-quality hosting of 300 million tiny files is hard. The average file size is just 450 byte. Dozens of tile servers have been written to tackle this problem, but they all have their limitations.
The original idea of this project is to avoid using tile servers altogether. Instead, the tiles are directly served from Btrfs partition images + hard links using an optimised nginx config. I wrote [extract_mbtiles](scripts/tile_gen/extract_mbtiles) and [shrink_btrfs](scripts/tile_gen/shrink_btrfs) scripts for this very purpose. The original idea of this project is to avoid using tile servers altogether. Instead, the tiles are directly served from Btrfs partition images + hard links using an optimised nginx config. I wrote [extract_mbtiles](scripts/tile_gen/scripts/extract_mbtiles.py) and [shrink_btrfs](scripts/tile_gen/scripts/shrink_btrfs.py) scripts for this very purpose.
This replaces a running service with a pure, file-system-level implementation. Since the Linux kernel's file caching is among the highest-performing and most thoroughly tested codes ever written, it delivers serious performance. This replaces a running service with a pure, file-system-level implementation. Since the Linux kernel's file caching is among the highest-performing and most thoroughly tested codes ever written, it delivers serious performance.
@@ -91,7 +91,7 @@ _note: Tile generation is 100% optional, as we are providing the processed full
The `tile_gen` script downloads a full planet OSM extract and runs it through Planetiler. The `tile_gen` script downloads a full planet OSM extract and runs it through Planetiler.
The created .mbtiles file is then extracted into a Btrfs partition image using the custom [extract_mbtiles](scripts/tile_gen/extract_mbtiles) script. The partition is shrunk using the [shrink_btrfs](scripts/tile_gen/shrink_btrfs) script. The created .mbtiles file is then extracted into a Btrfs partition image using the custom [extract_mbtiles](scripts/tile_gen/scripts/extract_mbtiles.py) script. The partition is shrunk using the [shrink_btrfs](scripts/tile_gen/scripts/shrink_btrfs.py) script.
Finally, it's uploaded to a public Cloudflare R2 bucket using rclone. Finally, it's uploaded to a public Cloudflare R2 bucket using rclone.
@@ -111,10 +111,9 @@ See [self hosting docs](docs/self_hosting.md).
### Full planet downloads ### Full planet downloads
You can directly download the processed full planet runs on the following URLs: You can directly download the processed full planet runs on the following URL patterns:
https://planet.openfreemap.com/20240607_232801_pt/tiles.mbtiles // 89 GB, mbtiles file https://planet.openfreemap.com/20240607_232801_pt/tiles.btrfs.gz // 86 GB
https://planet.openfreemap.com/20240607_232801_pt/tiles.btrfs.gz // 86 GB, Btrfs partition image
Replace the `20240607_232801_pt` part with any newer run, from the [index file](https://planet.openfreemap.com/index.txt). Replace the `20240607_232801_pt` part with any newer run, from the [index file](https://planet.openfreemap.com/index.txt).

5
scripts/tile_gen/tile_gen.py
View File

@@ -25,8 +25,9 @@ def make_tiles(area):
Generate tiles for a given area Generate tiles for a given area
""" """
# run_planetiler(area) run_folder = run_planetiler(area)
make_btrfs(Path('/data/ofm/tile_gen/runs/monaco/20240826_230406_pt')) make_btrfs(run_folder)
# make_btrfs(Path('/data/ofm/tile_gen/runs/monaco/20240826_230406_pt'))
@cli.command() @cli.command()

8
scripts/tile_gen/tile_gen_lib/extract.py
View File

@@ -66,6 +66,8 @@ def make_btrfs(run_folder: Path):
stderr=err, stderr=err,
) )
os.unlink('tiles.mbtiles')
shutil.copy('mnt_rw/extract/osm_date', '.') shutil.copy('mnt_rw/extract/osm_date', '.')
# process logs # process logs
@@ -122,15 +124,13 @@ def make_btrfs(run_folder: Path):
os.unlink('image.btrfs') os.unlink('image.btrfs')
shutil.move('image2.btrfs', 'tiles.btrfs') shutil.move('image2.btrfs', 'tiles.btrfs')
# parallel gzip # parallel gzip (pigz)
subprocess.run(['pigz', 'tiles.btrfs', '--fast'], check=True) subprocess.run(['pigz', 'tiles.btrfs', '--fast'], check=True)
# logs # move logs
Path('logs').mkdir() Path('logs').mkdir()
for pattern in ['*.log', '*.txt']: for pattern in ['*.log', '*.txt']:
for file in Path().glob(pattern): for file in Path().glob(pattern):
shutil.move(file, 'logs') shutil.move(file, 'logs')
print('extract_btrfs.py DONE') print('extract_btrfs.py DONE')
return run_folder

3
website/src/content/how_to_use/self_hosting.md
View File

@@ -1,3 +1,4 @@
## Self-hosting ## Self-hosting
You can also download our processed planet MBTiles and Btrfs images if you want to self-host yourself. Details can be found on [GitHub](https://github.com/hyperknot/openfreemap). You can also download our processed full planet Btrfs images if you want to self-host yourself. Details can be found on [GitHub](https://github.com/hyperknot/openfreemap).