NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

I have a lengthy osm2pgsql import that failed due to insufficient memory. I have added 16GB af cache and would like to resume the process if possible. Based on the following, is this possible or do I need to restart?

osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 2000 --number-processes 1 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/europe-latest.osm.pbf

osm2pgsql version 0.95.0-dev (64 bit id space) Using lua based tag processing pipeline with script /home/ge/src/openstreetmap-carto/openstreetmap-carto.lua Using projection SRS 3857 (Spherical Mercator) Setting up table: planet_osm_point Setting up table: planet_osm_line Setting up table: planet_osm_polygon Setting up table: planet_osm_roads Allocating memory for dense node cache Allocating dense node cache in one big chunk Allocating memory for sparse node cache Sharing dense sparse Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11 Mid: pgsql, cache=2000 Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Reading in file: /home/ge/data/europe-latest.osm.pbf Using PBF parser. Processing: Node(2034978k 40.7k/s) Way(248018k 0.15k/s) Relation(3846660 3.73/s) parse time: 2749907s Node stats: total(2034978773), max(5396860910) in 49970s Way stats: total(248018225), max(559553572) in 1668599s Relation stats: total(3846682), max(7999697) in 1031338s Committing transaction for planet_osm_point Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Committing transaction for planet_osm_roads Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Using lua based tag processing pipeline with script /home/ge/src/openstreetmap-carto/openstreetmap-carto.lua Killed

asked 16 Jul '18, 14:09

Jacknj's gravatar image

Jacknj
11112
accept rate: 0%


I would definitely switch to --flat-nodes if you run out of memory, as this will store the nodes on disk instead of RAM. Especially with SSD, you shouldn't worry about slow imports with --flat-nodes. I recently succeeded to load the whole of Europe on a 12GB RAM virtual machine with plenty of disk space, but only after I dropped inclusion of advanced settings like cache, and simply let osm2pgsql itself figure out its optimal processing setting. It took just some 17 hours on a Core i7 laptop.

See here for a few more details and observations:

https://forum.openstreetmap.org/viewtopic.php?id=62495

permanent link

answered 16 Jul '18, 17:48

mboeringa's gravatar image

mboeringa
1.5k21527
accept rate: 9%

Not easily, no.

permanent link
This answer is marked "community wiki".

answered 16 Jul '18, 15:51

SomeoneElse's gravatar image

SomeoneElse ♦
36.9k71370866
accept rate: 16%

wikified 16 Jul '18, 15:52

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×15
×1

question asked: 16 Jul '18, 14:09

question was seen: 4,422 times

last updated: 16 Jul '18, 17:48

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum