Populating database for Overpass application

Question

Hello.

Yesterday I've started installation of Overpass API on my dedicated server, following steps described here. Since yesterday I am waiting for database populating step to be finished. I've executed following actions:

Instaled Overpass app via tarball
Downloaded Europe data europe-latest.osm.bz2 from Geofabrik
Executed following command:

nohup init_osm3s.sh planet-latest.osm.bz2 $DB_DIR $EXEC_DIR &

I have encountered memory problems during this step and OOM killer interrupted population process. After investigation I've realized that my VPS had memory swaping disabled, so I've enabled memory swapping and set swapfile to 8GB.

After that I repeated step 3. once more, this time no problems showed up. But I am wondering if everything is ok, because it is now ~20 hours since population started, and logs in nohup.out are indicating only 9 nodes flushed into database:

Reading XML file ... elapsed node 130099515. Flushing to database ...... done.
Reading XML file ... elapsed node 270284771. Flushing to database ...... done.
Reading XML file ... elapsed node 299988093. Flushing to database ...... done.
Reading XML file ... elapsed node 337586538. Flushing to database ...... done.
Reading XML file ... elapsed node 366679358. Flushing to database ...... done.
Reading XML file ... elapsed node 421120908. Flushing to database ...... done.
Reading XML file ... elapsed node 472304225. Flushing to database ...... done.
Reading XML file ... elapsed node 523068300. Flushing to database ...... done.
Reading XML file ... elapsed node 580189749. Flushing to database ...... done.

My questions are: is the speed normal? I can imagine that europe has millions of nodes so if every node has to be flushed that way, it will take almost forever to complete. How can I speed-up that process?

My VM:

60 GB SSD
4GB of RAM, 8GB SSD swap space
1 vCore
Ubuntu 18 LTS

Currently all RAM memory is used and 4GB of swap space is utilized (actually no big changes in memory usage from yesterday).

If a larger machine is required then it is no problem, I can add more resources any time (Google compute engine here...). But I've tought i've met the requirements listed in wiki.

Accepted Answer

2

It doesn't list all the nodes, just gives you a node ID every now and then. You seem to have processed about 10% of data in 20 hours so the whole import would take a bit over a week. I think your RAM is probably the reason why it's going slow. If you can allocate resources dynamically, give it 32 GB for the import and you can go down to 8 GB again after that has finished.

permanent link

answered 24 Oct '19, 09:04

Frederik Ramm ♦
82.5k●92●720●1273
accept rate: 23%

Great! I will do that, thank You :) One more question: 60GB of disk space is enough? It seams like 80% is consumed just now. Should I upgrade disk space also (only europe needed)?

(24 Oct '19, 09:28) igoras1993

I have a machine with an europe-wide Overpass database and it uses ~ 120 GB.

(24 Oct '19, 09:43) Frederik Ramm ♦

Thank You very much!

(24 Oct '19, 09:50) igoras1993

Populating database for Overpass application

Follow this question

Related questions