# OpenFreeMap [openfreemap.org](https://openfreemap.org) ## What is OpenFreeMap? OpenFreeMap (OFM) provides free map hosting so you can display custom maps on your website and apps. It is truly **free**: there are no limits on the number of map views or requests you can make, nor on how you use your map. There is no registration page, user database, API keys, or cookies. It is truly **open-source**: everything, including the full production setup, is in this repo. Map data is from OpenStreetMap. ## Goals of this project The goal of this project is to provide free, production-quality vector-tile hosting using existing tools. Currently these tools are: [OpenStreetMap](https://www.openstreetmap.org/copyright), [OpenMapTiles](https://github.com/openmaptiles/openmaptiles), [Planetiler](https://github.com/onthegomap/planetiler) , [MapLibre](https://maplibre.org/) and [Natural Earth](https://www.naturalearthdata.com/) and soon [tilemaker](https://github.com/systemed/tilemaker). OFM does not want to be an alternative to any of these projects. If the community decides, we can replace any of these tools. The scope of this repo is limited (see below). Once we figure out the technical details, ideally, there should be few commits here, while everything keeps working: the map tiles are automatically generated, HTTP servers are automatically updated and load balancing takes care of failing servers. The [styles repo](https://github.com/hyperknot/openfreemap-styles) - on the other hand - is a possibly never ending development. Contributions are more than welcome! ### Limitations of this project The only way this project can possibly work is to be super focused about what it is and what it isn't. OFM has the following limitations by design. 1. OFM is not providing: - search or geocoding - route calculation, navigation or directions - static image generation - raster tile hosting - satellite image hosting - elevation lookup - custom tile or dataset hosting 2. OFM is not something you can install on your dev machine. OFM is a deploy script specifically made to set up clean Ubuntu servers or virtual machines. It uses [Fabric](https://www.fabfile.org/) and runs commands over SSH. With a single command it can set up a production-ready OFM server, both for tile hosting and generation. This repo is also Docker free. If someone wants to make a Docker-based version of this, I'm more than happy to link it here. 3. OFM does not promise worry-free automatic updates for self-hosters. Only enable the cron job if you keep a close eye on this repo. ## Code structure The project has the following parts #### deploy server - ssh_lib and init-server.py This sets up everything on a clean Ubuntu server. You run it locally and it sets up the server via SSH. You specify `--tile-gen` and/or `--http-host` at startup. #### tile generation - scripts/tile_gen The `tile_gen` scripts download a full planet OSM extract and run it through Planetiler (and soon tilemaker). Currently a run is triggered manually, by running `planetiler_{area}.sh`. The created .mbtiles file is then extracted into a BTRFS partition image using the custom [extract_mbtiles](scripts/tile_gen/extract_mbtiles) script. The partition is shrunk using the [shrink_btrfs](scripts/tile_gen/shrink_btrfs) script. Finally, it's uploaded to a public Cloudflare R2 bucket using rclone. *Note: Perhaps the most original aspect of this repository is the use of partition images and hard links. I experimented with ext4 first, but BTRFS proved to be a better fit for the job, with much smaller resulting images. I wrote extract_mbtiles and shrink_btrfs scripts for this very purpose.* #### HTTP host - scripts/http_host Inside `http_host`, all work is done by `host_manager.py`. It checks the most up-to-date files in the public buckets and downloads/extracts them locally, if needed. It mounts the downloaded BTRFS images in `/mnt/ofm`, creates the correct TileJSON file and updates nginx with the correct config. You can run `./host_manager.py --help` to see which options are available. Some commands can be run locally, including on non-linux machines. #### styles - [styles repo](https://github.com/hyperknot/openfreemap-styles) A very important part, probably needs the most work in the long term future. ## How to run? Use Python 3.10/3.11. Create virtualenv using: `source prepare-virtualenv.sh` It's recommended to use [direnv](https://direnv.net/), to have automatic venv activation. ##### 1. Prepare config folder 1. copy the .sample files and change the values 2. SSH_PASSWD is only needed if you don't use SSH keys. 3. rclone.conf is only needed for uploading. For http_host there is no need for this file. 4. certs - these are the certs for nginx. If you put a cert here, it'll be uploaded to `/data/nginx/certs`. Currently the nginx config is hard coded to use for `openfreemap.org.cert` and `openfreemap.org.key`. ##### 2. Deploy a HTTP host You run the deploy script locally, and it'll connect to an SSH server, like this `./init-server.py HOSTNAME --http-host` After this, go for a walk and by the time you come back it should be up and running. When it's finished it's a good idea to delete the cron job with `rm /etc/cron.d/ofm_http_host` , see warning below. ##### 3. Deploy tile gen server (optional) - If you have a beefy machine and you want to generate tiles yourselfm you can run the same script with `--tile-gen`. You generally don't need this since we provide already processed tile downloads for free. #### Warning This project is made to run on clean servers or virtual machines dedicated for this project. The scripts need sudo permissions as they mount/unmount disk images. Do not run this on your dev machine without using virtual machines. If you do, please make sure you understand exactly what each script is doing. ## Downloads and buckets There are three public buckets: - https://assets.openfreemap.com - contains fonts, sprites, styles, versions. index: [dirs](https://assets.openfreemap.com/dirs.txt), [files](https://assets.openfreemap.com/index.txt) - https://planet.openfreemap.com - full planet runs. index: [dirs](https://planet.openfreemap.com/dirs.txt), [files](https://planet.openfreemap.com/index.txt) - https://monaco.openfreemap.com - identical runs to the full planet, but only for Monaco area. Very tiny, ideal for development. index: [dirs](https://monaco.openfreemap.com/dirs.txt), [files](https://monaco.openfreemap.com/index.txt) #### Full planet downloads You can directly download the processed full planet runs on the following URLs: https://planet.openfreemap.com/20231221_134737_pt/tiles.mbtiles // 84 GB, mbtiles file https://planet.openfreemap.com/20231221_134737_pt/tiles.btrfs.gz // 81 GB, BTRFS partition image Replace the `20231221_134737_pt` part with any newer run, from the [index file](https://planet.openfreemap.com/index.txt). ## HTTPS certs The current HTTPS system is made to use long term Cloudflare origin certificates. The same certificates are uploaded to all the server. This is only possible because CF certs are valid for 15 years. Once Load Balancing on CF is working, next step will be to integrate Let's Encrypt. If you know how to do this, please comment in the Discussions. ## Domains and Cloudflare The project has two domains: .org and .com. Currently, both are on Cloudflare. The general public only interacts with the .org domain. It has been designed so that this domain can be migrated away from Cloudflare if needed. The .com domain hosts the R2 buckets, which are required to be on Cloudflare. This domain will always remain on CF. ## FAQ ### System requirements Ubuntu 22+ Disk space: about 240 GB for hosting a single run, 500 GB for tile gen. ### What about PMTiles? I would have loved to use PMTiles; they are a brilliant idea! Unfortunately, making range requests in 80 GB files just doesn't work in production. It is fine for files smaller than 500 MB, but it has terrible latency and caching issues for full planet datasets. If PMTiles implements splitting to <10 MB files, it can be a valid alternative to running servers. ## Contributing Contributors welcome! Smaller tasks: - Add tilemaker, so we see the difference between planetiler and tilemaker and they can both validate their output based on this comparison. - Figure out how to use Let's Encrypt on multiple servers with Round Robin DNS. - Cloudflare worker for indexing the public buckets, instead of manually generating index.txt files. - Some of the POI icons are missing in the styles. Bigger tasks: - Split the styles to blocks and build them up from blocks. For example, there should be a POI block, a label block, a road-style related block. - Implement automatic updates for tile gen, uploading, testing and setting versions. Tasks outside the scope of this project: - Make a successor for the OpenMapTiles schema. - Docker image for running this self-hosted on any machine. ## Changelog v0.1 - everything works. 1 server for tile gen, 1 server for HTTP host. <- we are here! ## License The license of this project is [MIT](https://www.tldrlegal.com/license/mit-license). Map data is from [OpenStreetMap](https://www.openstreetmap.org/copyright). The licenses for included projects are listed in [LICENSE.md](https://github.com/hyperknot/openfreemap/blob/main/LICENSE.md).