Hello.

Yesterday I've started installation of Overpass API on my dedicated server, following steps described here. Since yesterday I am waiting for database populating step to be finished. I've executed following actions:

  1. Instaled Overpass app via tarball
  2. Downloaded Europe data europe-latest.osm.bz2 from Geofabrik
  3. Executed following command:

nohup init_osm3s.sh planet-latest.osm.bz2 $DB_DIR $EXEC_DIR &

I have encountered memory problems during this step and OOM killer interrupted population process. After investigation I've realized that my VPS had memory swaping disabled, so I've enabled memory swapping and set swapfile to 8GB.

After that I repeated step 3. once more, this time no problems showed up. But I am wondering if everything is ok, because it is now ~20 hours since population started, and logs in nohup.out are indicating only 9 nodes flushed into database:

Reading XML file ... elapsed node 130099515. Flushing to database ...... done.
Reading XML file ... elapsed node 270284771. Flushing to database ...... done.
Reading XML file ... elapsed node 299988093. Flushing to database ...... done.
Reading XML file ... elapsed node 337586538. Flushing to database ...... done.
Reading XML file ... elapsed node 366679358. Flushing to database ...... done.
Reading XML file ... elapsed node 421120908. Flushing to database ...... done.
Reading XML file ... elapsed node 472304225. Flushing to database ...... done.
Reading XML file ... elapsed node 523068300. Flushing to database ...... done.
Reading XML file ... elapsed node 580189749. Flushing to database ...... done.

My questions are: is the speed normal? I can imagine that europe has millions of nodes so if every node has to be flushed that way, it will take almost forever to complete. How can I speed-up that process?

My VM:

  1. 60 GB SSD
  2. 4GB of RAM, 8GB SSD swap space
  3. 1 vCore
  4. Ubuntu 18 LTS

Currently all RAM memory is used and 4GB of swap space is utilized (actually no big changes in memory usage from yesterday).

If a larger machine is required then it is no problem, I can add more resources any time (Google compute engine here...). But I've tought i've met the requirements listed in wiki.

asked 24 Oct, 08:56

igoras1993's gravatar image

igoras1993
163
accept rate: 0%


It doesn't list all the nodes, just gives you a node ID every now and then. You seem to have processed about 10% of data in 20 hours so the whole import would take a bit over a week. I think your RAM is probably the reason why it's going slow. If you can allocate resources dynamically, give it 32 GB for the import and you can go down to 8 GB again after that has finished.

permanent link

answered 24 Oct, 09:04

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
71.3k846451113
accept rate: 24%

Great! I will do that, thank You :) One more question: 60GB of disk space is enough? It seams like 80% is consumed just now. Should I upgrade disk space also (only europe needed)?

(24 Oct, 09:28) igoras1993

I have a machine with an europe-wide Overpass database and it uses ~ 120 GB.

(24 Oct, 09:43) Frederik Ramm ♦

Thank You very much!

(24 Oct, 09:50) igoras1993
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×355
×92

question asked: 24 Oct, 08:56

question was seen: 166 times

last updated: 24 Oct, 09:50

powered by OSQA