NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

I've previously imported a single country's pbf file using osm2pgsql. I'm now attempting this with the whole of europe.

I'm doing this on an UBUNTU VM with 2 cores and 8GB RAM. The host is an i7 930 quadcore clocked to 4000Mhz with 16GB RAM

Things were going well, it processed the nodes, and ways and had just started on the relations - was up to about 10k I think last check. This had taken about a week. That was 3 days ago, since then the VM has been running at ~100% CPU - the host is clocking between 35-50% utilization from the VM. I cannot get the unlock prompt to appear on the VM, so I cannot get in to see the console output.

The VM still appears active - no error messages, and the vmx file is periodically being updated. ResourceMonitor on the host, shows the VM is reading from it's disk at around 40Mb/sec and writing at around 5kb/sec. I'm loathe to kill it and potentially waste nearly 2 weeks of importing.

Does anyone know if this is normal?

Is processing the relations more CPU intensive than the nodes and ways?

Has anyone got any idea of the relative lengths of the processes of importing nodes, ways and relations? (Obviously the absolute values will depend on hardware.)

asked 18 Jun '13, 15:34

Tangotonyb's gravatar image

Tangotonyb
51337
accept rate: 0%

edited 18 Jun '13, 15:46

SomeoneElse's gravatar image

SomeoneElse ♦
36.9k71370866

1

what are the parameters with which you imported?

(18 Jun '13, 19:01) apmon
1

I think: osm2pgsql --slim -d gis -C 1500 --number-processes 2 ~/planet/planet-latest.osm.pbf - not sure about cache-size, tried with 6000 and it failed, again with 5000 and it failed, both after running for over a day, so I played safe and dropped it to under 3000

(18 Jun '13, 20:23) Tangotonyb
2

Bah - came back to life for a day, and then process got killed with 900k relations imported. That's with 2000 cache.

(20 Jun '13, 15:07) Tangotonyb

It is possible that the relations are meanwhile completed and the system is building the indexes which can take quite a long time and lead to intense disk I/O which might well be very taxing for your VM.

permanent link

answered 18 Jun '13, 15:58

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
82.5k927201273
accept rate: 23%

2

Thanks - that makes sense - I'd have expected the relations to have completed - the intense disk I/O is exactly what I'm seeing. I'll leave it a few days to see what happens.

(18 Jun '13, 18:39) Tangotonyb
2

It is also possible that it has moved into the "going over pending ways" phase, which pulls in a list of all pending way ids (typically about half of all ways), which might have pushed you into using swap.

(18 Jun '13, 19:04) apmon

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×44

question asked: 18 Jun '13, 15:34

question was seen: 3,400 times

last updated: 20 Jun '13, 15:07

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum