NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

I've started a full osm planet import 6 days ago with:

osm2pgsql --number-processes 8 --slim --cache 14384 -d osm_planet_db --hstore --hstore-match-only -W -m -S default.style osm-planet-150303.osm.pbf

It is now at:

Processing: Node(2786658k 90.9k/s) Way(243561k 0.52k/s) Relation(0 0.00/s)

So nodes have been processed, ways almost, relations not yet.

Is it possible to estimate how more days it will take to finalize?

2nd point: I followed the instruction for optimizing PostgreSQL (set autovacuum off, incresed checkpoint_segments, etc.). But I only now read that my --cache setting of 14384 might be a too low for a full planet import:

"osm2pgsql relies much on its node cache during import. If the nodes do not fit into the cache in slim mode it needs to do database lookups which slow down the process. (Without slim mode, it fails if the nodes do not fit in the cache). Use enough cache so all nodes are cached. -C 22000 seems to do the job, even if that means you have to configure more swap space."

Does it make sense to kill the osm2pqsql process and restart with --cache 28000? (I have a total of 32GB RAM, 8 cores, 64bit CentOS). That is, throw away the last 6 days of importing because it will still take ages to finish, and with the increased Cache it will still be faster in the end?

asked 08 Apr '15, 09:16

jnachtigall's gravatar image

jnachtigall
101448
accept rate: 0%


You're likely to get a performance of under 5 relations per second on this machine, which would mean at least a week for relation processing, and after that you still need to go through index creation. My guess is that you're looking for at least two more weeks here. You can speed up the job with more cache and the --flatnodes option but probably not by 30%.

The fastest way to get a planet file imported is to order a large SSD disk and install it to your server. If you order it now, receive it tomorrow, install it a day later, and then restart the import, you can be done within a week.

permanent link

answered 08 Apr '15, 10:45

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
82.5k927201273
accept rate: 23%

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×196
×165
×134
×25

question asked: 08 Apr '15, 09:16

question was seen: 7,621 times

last updated: 08 Apr '15, 10:45

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum