work

README.md

@@ -41,7 +41,7 @@ The only way this project can possibly work is to be super focused about what it
 
    This repo is also Docker free. If someone wants to make a Docker-based version of this, I'm more than happy to link it here.
 
-3. OFM does not promise worry-free automatic updates for self-hosters. Only enable the cron job if you keep a close eye on this repo.
+3. OFM does not promise worry-free automatic updates for self-hosters. Only use the autoupdate command if you keep a close eye on this repo.
 
 ## Code structure
 
@@ -70,7 +70,7 @@ You can run `./host_manager.py --help` to see which options are available. Some
 
 *note: Tile generation is 100% optional, as we are providing the processed full planet files for public download.*
 
-The `tile_gen` scripts downloads a full planet OSM extract and runs it through Planetiler (or soon tilemaker). Currently a run is triggered manually, by running `planetiler_{area}.sh`.
+The `tile_gen` scripts downloads a full planet OSM extract and runs it through Planetiler (or soon tilemaker).
 
 The created .mbtiles file is then extracted into a BTRFS partition image using the custom [extract_mbtiles](scripts/tile_gen/extract_mbtiles) script. The partition is shrunk using the [shrink_btrfs](scripts/tile_gen/shrink_btrfs) script.
 
@@ -80,48 +80,11 @@ Finally, it's uploaded to a public Cloudflare R2 bucket using rclone.
 
 A very important part, probably needs the most work in the long term future.
 
-## How to run?
-
-*note: For most users, **you don't need to run anything**! The tiles are provided free of charge, without registration. Read the "How can I use it?" section on https://openfreemap.org*
-
-The instructions below are intended only for those who have a large server and would like to self-host.
-
-Use Python 3.10/3.11. 
-
-Create virtualenv using: `source prepare-virtualenv.sh`
-
-It's recommended to use [direnv](https://direnv.net/), to have automatic venv activation.
-
-*note: Currently the domains are hard coded, so you'll need to edit the nginx templates using a text editor or sed.*
-
-##### 1. Prepare config folder
-
-1. copy the .sample files and change the values
-
-2. SSH_PASSWD is only needed if you don't use SSH keys.
-
-3. rclone.conf is only needed for tile generation. For http_host there is no need for this file.
-
-4. certs - used by nginx for HTTPS, they are uploaded to `/data/nginx/certs`.
 
 
-##### 2. Deploy a HTTP host
+## Self hosting
 
-You run the deploy script locally, and it'll connect to an SSH server, like this
+See [self hosting docs](docs/self_hosting.md).
-
-`./init-server.py HOSTNAME --http-host`
-
-After this, go for a walk and by the time you come back it should be up and running.
-
-When it's finished it's a good idea to delete the cron job with `rm /etc/cron.d/ofm_http_host` , see warning below.
-
-##### 3. Deploy tile gen server (optional)
-
-- If you have a beefy machine and you want to generate tiles yourself, you can run the same script with `--tile-gen`. You generally don't need this since we provide already processed tile downloads for free.
-
-#### Warning
-
-This project is made to run on clean servers or virtual machines dedicated for this project. The scripts need sudo permissions as they mount/unmount disk images. Do not run this on your dev machine without using virtual machines. If you do, please make sure you understand exactly what each script is doing.
 
 
 
@@ -139,12 +102,6 @@ I run some [benchmarks](docs/quick_notes/http_benchmark.md) on a Hetzner server,
 
 ## FAQ
 
-### System requirements
-
-Ubuntu 22+
-
-Disk space: about 240 GB for hosting a single run, 500 GB for tile gen.
-
 ### Full planet downloads
 
 You can directly download the processed full planet runs on the following URLs:
@@ -154,6 +111,14 @@ https://planet.openfreemap.com/20231221_134737_pt/tiles.btrfs.gz // 81 GB, BTRFS
 
 Replace the `20231221_134737_pt` part with any newer run, from the [index file](https://planet.openfreemap.com/index.txt).
 
+### Public buckets
+
+There are three public buckets:
+
+- https://assets.openfreemap.com - contains fonts, sprites, styles, versions. index: [dirs](https://assets.openfreemap.com/dirs.txt), [files](https://assets.openfreemap.com/index.txt)
+- https://planet.openfreemap.com - full planet runs. index: [dirs](https://planet.openfreemap.com/dirs.txt), [files](https://planet.openfreemap.com/index.txt)
+- https://monaco.openfreemap.com - identical runs to the full planet, but only for Monaco area. Very tiny, ideal for development. index: [dirs](https://monaco.openfreemap.com/dirs.txt), [files](https://monaco.openfreemap.com/index.txt)
+
 ### HTTPS certs
 
 The current HTTPS system is made to use long term Cloudflare origin certificates. The same certificates are uploaded to all the servers. This is only possible because CF certs are valid for 15 years.
@@ -168,14 +133,6 @@ The general public only interacts with the .org domain. It has been designed so
 
 The .com domain hosts the R2 buckets, which are required to be on Cloudflare. This domain will always remain on CF.
 
-### Public buckets
-
-There are three public buckets:
-
-- https://assets.openfreemap.com - contains fonts, sprites, styles, versions. index: [dirs](https://assets.openfreemap.com/dirs.txt), [files](https://assets.openfreemap.com/index.txt)
-- https://planet.openfreemap.com - full planet runs. index: [dirs](https://planet.openfreemap.com/dirs.txt), [files](https://planet.openfreemap.com/index.txt)
-- https://monaco.openfreemap.com - identical runs to the full planet, but only for Monaco area. Very tiny, ideal for development. index: [dirs](https://monaco.openfreemap.com/dirs.txt), [files](https://monaco.openfreemap.com/index.txt)
-
 ### What about PMTiles?
 
 I would have loved to use PMTiles; they are a brilliant idea!

docs/self_hosting.md

@@ -0,0 +1,97 @@
+# Self-hosting Howto
+
+*note: For most users, **you don't need to run anything**! The tiles are hosted free of charge, without registration. Read the "How can I use it?" section on https://openfreemap.org*
+
+
+
+When self-hosting, there are two tasks you can set up on a server (see details in the repo README).
+
+- **http-host**
+
+- **tile-gen**
+
+note: Tile generation is 100% optional, as we are providing the processed full planet files for public download. It also requires a beefy machine, see below.
+
+### System requirements
+
+##### Disk space
+
+​	**http-host**: 300 GB for hosting a single run
+
+​	**tile-gen**: 500 GB for 
+
+##### RAM
+
+​	**http-host**: 4 GB
+
+​	**tile-gen**: 64 GB+ RAM.
+
+##### OS
+
+​	**Ubuntu 22+**
+
+
+
+### Limitations
+
+There is two limitation in the current beta version:
+
+- You have to set up Let's Encrypt manually or supply your certs.
+
+- The domain is hard-coded to `tiles.openfreemap.org` - you have to edit this.
+
+
+
+---
+
+### Warning
+
+This project is made to run on clean servers or virtual machines dedicated for this project. The scripts need sudo permissions as they mount/unmount disk images. Do not run this on your dev machine without using virtual machines. If you do, please make sure you understand exactly what each script is doing.
+
+---
+
+
+
+
+## Instructions
+
+Create virtualenv using: `source prepare-virtualenv.sh` 
+
+It's recommended to use [direnv](https://direnv.net/), to have automatic venv activation.
+
+#### 1. Prepare `config` folder
+
+1. If you are not using SSH keys, copy `.env.sample` to `.env` and set the password.
+1. `certs` - The contents of this folder gets uploaded to `/data/nginx/certs`.
+1. If you want to run tile generation and upload via rclone, you can copy the `rclone.conf.sample` file as well. For simple self-hosting there is no need for this.
+
+#### 2. Certs and domains
+
+Currently the domain is hard coded to `tiles.openfreemap.org`. Please search & replace for this string to your custom one.
+
+The script is made with long expiry CloudFlare origin certificates in mind, which are placed in the `config/certs` folder. For self-hosting you may want to use Let's Encrypt or similar automated tool. The support for this is coming later.
+
+
+#### 3. Deploy a http-host
+
+You run the deploy script locally. It'll connect to an SSH server, like this
+
+`./init-server.py http-host-once HOSTNAME`
+
+After this, go for a walk and by the time you come back it should be up and running with the latest planet tiles deployed. Don't worry about the "Download aborted" lines in the meanwhile, it's a bug in CloudFlare.
+
+#### 4. Deploy tile-gen server (optional)
+
+If you have a really beefy machine (see above) and you want to generate tiles yourself, you can run `./init-server.py tile-gen HOSTNAME`. 
+
+Trigger a run manually, by running `planetiler_{area}.sh`. Recommended to use tmux or similar, as it can take days.
+
+
+
+
+
+### HTTPS certs
+
+The current HTTPS system is made to use long term Cloudflare origin certificates. The same certificates are uploaded to all the servers. This is only possible because CF certs are valid for 15 years.
+
+Once Load Balancing on CF is working, next step will be to integrate Let's Encrypt. If you know how to do this, please comment in the Discussions.

init-server.py

@@ -141,6 +141,8 @@ def get_connection(hostname, user, port):
     ssh_passwd = dotenv_values(f'{CONFIG_DIR}/.env').get('SSH_PASSWD')
 
     if ssh_passwd:
+        print('Using SSH password')
+
         c = Connection(
             host=hostname,
             user=user,
@@ -178,10 +180,10 @@ def http_host_once(hostname, user, port):
         return
 
     c = get_connection(hostname, user, port)
-
     prepare_shared(c)
 
     prepare_http_host(c)
+    run_http_host_sync(c)
 
 
 @cli.command()
@@ -191,7 +193,6 @@ def http_host_autoupdate(hostname, user, port):
         return
 
     c = get_connection(hostname, user, port)
-
     prepare_shared(c)
 
     prepare_http_host(c)
@@ -205,7 +206,6 @@ def tile_gen(hostname, user, port):
         return
 
     c = get_connection(hostname, user, port)
-
     prepare_shared(c)
 
     prepare_tile_gen(c)
@@ -215,9 +215,9 @@ def tile_gen(hostname, user, port):
 @common_options
 def debug(hostname, user, port):
     c = get_connection(hostname, user, port)
-
+    c.run('pwd')
-    upload_https_host_files(c)
+    # upload_https_host_files(c)
-    run_http_host_sync(c)
+    # run_http_host_sync(c)
 
 
 if __name__ == '__main__':

scripts/http_host/http_host_lib/download_tileset.py

@@ -40,6 +40,7 @@ def download_and_extract_tileset(area: str, version: str, runs_dir: Path) -> boo
     target_file = temp_dir / 'tiles.btrfs.gz'
     download_file_aria2(url, target_file)
 
+    print('Uncompressing...')
     subprocess.run(['unpigz', temp_dir / 'tiles.btrfs.gz'], check=True)
     btrfs_src = temp_dir / 'tiles.btrfs'