This is a static archive of our old OpenStreetMap Help Site. Please post any new questions and answers at community.osm.org.

Can I resume a osm2pgsql import after a out-of-memory-kill?

0

I have a lengthy osm2pgsql import that failed due to insufficient memory. I have added 16GB af cache and would like to resume the process if possible. Based on the following, is this possible or do I need to restart?

osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2000 --number-processes 1 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/europe-latest.osm.pbf
osm2pgsql version 0.95.0-dev (64 bit id space)
Using lua based tag processing pipeline with script /home/ge/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
Mid: pgsql, cache=2000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /home/ge/data/europe-latest.osm.pbf
Using PBF parser.
Processing: Node(2034978k 40.7k/s) Way(248018k 0.15k/s) Relation(3846660 3.73/s)  parse time: 2749907s
Node stats: total(2034978773), max(5396860910) in 49970s
Way stats: total(248018225), max(559553572) in 1668599s
Relation stats: total(3846682), max(7999697) in 1031338s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using lua based tag processing pipeline with script /home/ge/src/openstreetmap-carto/openstreetmap-carto.lua
Killed

asked 16 Jul '18, 14:09

Jacknj's gravatar image

Jacknj
11112
accept rate: 0%


2 Answers:

2

I would definitely switch to --flat-nodes if you run out of memory, as this will store the nodes on disk instead of RAM. Especially with SSD, you shouldn't worry about slow imports with --flat-nodes. I recently succeeded to load the whole of Europe on a 12GB RAM virtual machine with plenty of disk space, but only after I dropped inclusion of advanced settings like cache, and simply let osm2pgsql itself figure out its optimal processing setting. It took just some 17 hours on a Core i7 laptop.

See here for a few more details and observations:

https://forum.openstreetmap.org/viewtopic.php?id=62495

answered 16 Jul '18, 17:48

mboeringa's gravatar image

mboeringa
1.5k21527
accept rate: 9%

0

Not easily, no.

This answer is marked "community wiki".

answered 16 Jul '18, 15:51

SomeoneElse's gravatar image

SomeoneElse ♦
36.9k71370866
accept rate: 16%

wikified 16 Jul '18, 15:52

Source code available on GitHub .