Restarting osm2pqsql full planet import because of too low --cache setting? (import runs for 6 days already)

Question

I've started a full osm planet import 6 days ago with:

osm2pgsql --number-processes 8 --slim --cache 14384 -d osm_planet_db --hstore --hstore-match-only -W -m -S default.style osm-planet-150303.osm.pbf

It is now at:

Processing: Node(2786658k 90.9k/s) Way(243561k 0.52k/s) Relation(0 0.00/s)

So nodes have been processed, ways almost, relations not yet.

Is it possible to estimate how more days it will take to finalize?

2nd point: I followed the instruction for optimizing PostgreSQL (set autovacuum off, incresed checkpoint_segments, etc.). But I only now read that my --cache setting of 14384 might be a too low for a full planet import:

"osm2pgsql relies much on its node cache during import. If the nodes do not fit into the cache in slim mode it needs to do database lookups which slow down the process. (Without slim mode, it fails if the nodes do not fit in the cache). Use enough cache so all nodes are cached. -C 22000 seems to do the job, even if that means you have to configure more swap space."

Does it make sense to kill the osm2pqsql process and restart with --cache 28000? (I have a total of 32GB RAM, 8 cores, 64bit CentOS). That is, throw away the last 6 days of importing because it will still take ages to finish, and with the increased Cache it will still be faster in the end?

Accepted Answer

You're likely to get a performance of under 5 relations per second on this machine, which would mean at least a week for relation processing, and after that you still need to go through index creation. My guess is that you're looking for at least two more weeks here. You can speed up the job with more cache and the --flatnodes option but probably not by 30%.

The fastest way to get a planet file imported is to order a large SSD disk and install it to your server. If you order it now, receive it tomorrow, install it a day later, and then restart the import, you can be done within a week.

Restarting osm2pqsql full planet import because of too low --cache setting? (import runs for 6 days already)

Follow this question

Related questions