osm2pgsql import getting killed

Question

When I try to import planet-latest.osm.pbf I get the following error:

/usr/local/bin/osm2pgsql --create --flat-nodes --slim --username postgres --password --hstore --hstore-match-only --number-processes 6 --cache 50000 --style default.style --input-reader pbf planet-latest.osm.pbf
osm2pgsql SVN version 0.89.0-dev (64 bit id space)

Using built-in tag processing pipeline  
Using projection SRS 900913 (Spherical Mercator)  
Setting up table: planet_osm_point  
Setting up table: planet_osm_line  
Setting up table: planet_osm_polygon  
Setting up table: planet_osm_roads  
Allocating memory for dense node cache  
Allocating dense node cache in one big chunk  
Allocating memory for sparse node cache  
Sharing dense sparse  
Node-cache: cache=50000MB, maxblocks=800000*65536, allocation method=3  
Mid: Ram, scale=100  
Reading in file: planet-latest.osm.pbf  
Using PBF parser.  
Processing: Node(3095464k 2705.8k/s) Way(53703k 29.11k/s) Relation(0 0.00/s)Killed

Here is the message I get from dmesg:

Out of memory: Kill process 23636 (osm2pgsql) score 953 or sacrifice child  
Killed process 23636 (osm2pgsql) total-vm:75687764kB, anon-rss:60639884kB, file-rss:0kB

do I need more that 50GB RAM to import the planet file?

Accepted Answer

4

To import the whole planet, you need to use slim mode, but with your command line you are saving the flat nodes in a file called --slim. Add a file name after --flat-nodes.

Also, -C 28000 is plenty of cache to store all nodes in RAM.

permanent link

answered 13 Nov '15, 22:33

pnorman
2.4k●5●21●40
accept rate: 18%

Answer 2

1

--cache 50000 means you are assigning 50000 MB (50 GB) of your memory as node cache. This is too high when having "only" 50 GB of RAM. I'm not familiar with osm2pgsql so I can't suggest a good value for this option, but try to assign less than all your available RAM to it.

permanent link

answered 12 Nov '15, 15:51

scai ♦
33.3k●21●309●459
accept rate: 23%

Actually I have 60GB RAM, and I assigned 50GB. I also tried with 40GB, same thing happened!

(12 Nov '15, 15:58) khamooshi

... and I think (based on the fact that you've asked other questions about rendering) that other (smaller) imports have worked OK in the past?

(12 Nov '15, 16:36) SomeoneElse ♦

I didn't set up the other severs, so I don't know how they did it! btw, I use --flat-nodes and different projection.

(13 Nov '15, 08:42) khamooshi

osm2pgsql import getting killed

Follow this question

Related questions