|
I've previously imported a single country's pbf file using osm2pgsql. I'm now attempting this with the whole of europe. I'm doing this on an UBUNTU VM with 2 cores and 8GB RAM. The host is an i7 930 quadcore clocked to 4000Mhz with 16GB RAM Things were going well, it processed the nodes, and ways and had just started on the relations - was up to about 10k I think last check. This had taken about a week. That was 3 days ago, since then the VM has been running at ~100% CPU - the host is clocking between 35-50% utilization from the VM. I cannot get the unlock prompt to appear on the VM, so I cannot get in to see the console output. The VM still appears active - no error messages, and the vmx file is periodically being updated. ResourceMonitor on the host, shows the VM is reading from it's disk at around 40Mb/sec and writing at around 5kb/sec. I'm loathe to kill it and potentially waste nearly 2 weeks of importing. Does anyone know if this is normal? Is processing the relations more CPU intensive than the nodes and ways? Has anyone got any idea of the relative lengths of the processes of importing nodes, ways and relations? (Obviously the absolute values will depend on hardware.) |
|
It is possible that the relations are meanwhile completed and the system is building the indexes which can take quite a long time and lead to intense disk I/O which might well be very taxing for your VM. 2
Thanks - that makes sense - I'd have expected the relations to have completed - the intense disk I/O is exactly what I'm seeing. I'll leave it a few days to see what happens.
(18 Jun '13, 18:39)
Tangotonyb
2
It is also possible that it has moved into the "going over pending ways" phase, which pulls in a list of all pending way ids (typically about half of all ways), which might have pushed you into using swap.
(18 Jun '13, 19:04)
apmon
|
what are the parameters with which you imported?
I think: osm2pgsql --slim -d gis -C 1500 --number-processes 2 ~/planet/planet-latest.osm.pbf - not sure about cache-size, tried with 6000 and it failed, again with 5000 and it failed, both after running for over a day, so I played safe and dropped it to under 3000
Bah - came back to life for a day, and then process got killed with 900k relations imported. That's with 2000 cache.