NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Whenever I try to import the planet.osm.bz2 or california.osm.bz2, I get this:

bash-4.2$ ./osm2pgsql -S ./default.style ../california.osm
osm2pgsql SVN version 0.80.0 (32bit id space)

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Mid: Ram, scale=100

Reading in file: ../california.osm
Processing: Node(41300k 56.5k/s) Way(0k 0.00k/s) Relation(0 0.00/s)Killed

asked 05 Oct '11, 18:08

NDFobia1158's gravatar image

NDFobia1158
61447
accept rate: 0%

edited 10 Oct '12, 10:40

SomeoneElse's gravatar image

SomeoneElse ♦
36.9k71370866


I would guess this is the "out of memory killer" shooting your process; type dmesg after the incident to see the latest system messages that would corrobate this. I am surprised that you should run out of memory on a file as small as the California one but maybe other processes are eating memory too? Try the --slim option on osm2pgsql which will slow down things but use less memory.

permanent link

answered 05 Oct '11, 18:20

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
82.5k927201273
accept rate: 23%

1

I run into the same error by using the --slim option. There are any other solutions?

(10 Oct '12, 09:25) marco-2012

Have you looked at the switch2osm.org site? While for smaller extracts you can use quite a small machine, you should take the systems requirements for a full planet import seriously.

(10 Oct '12, 09:41) SimonPoole ♦

Yes, I used switch2osm.org/serving-tiles/manually-building-a-tile-server-12-04/ to set up a tile server on a virtual machine. I can import download.geofabrik.de/openstreetmap/europe/germany/hamburg.osm.pbf successfully and everything works fine. But if I try to import download.geofabrik.de/openstreetmap/europe/germany.osm.pbf I get the "out of memory killer". The virtual machine has 1 GB RAM. Is it to small?

(10 Oct '12, 09:57) marco-2012

I suspect yes, Germany is something over 5% of all data. Very roughly I would suspect that you need at least 1GB just for the import process.

(10 Oct '12, 10:13) SimonPoole ♦

Just another data point / anecdote - I struggled to load england (1/6 the size of germany) on a VM allocated 2Gb without fiddling with settings, so I'd have thought that 1Gb was too small.

If buying more memory isn't an option, perhaps experiment with VM memory settings - monitor what's being used, try larger or smaller -C values, and try either swap within the VM or a larger memory usage (relying on swap on the host) to see what works best.

(10 Oct '12, 10:39) SomeoneElse ♦
1

I increased the RAM to 2 GB and tried to increase and decrease the -C value, but I still got an error (not sure what, but no out of memory error). After I leaved the -C value (I don't know what the default -C value is) it works without any errors. Thanks for your help.

(10 Oct '12, 13:49) marco-2012
showing 5 of 6 show 1 more comments

I had the same problem with canada-latest.pbf. (2gb file size) Using ksysguard I watched the osm2pgsql process consume first memory (4gb) and then swap (4gb) until all was gone and the process killed. To fix: su dd if=/dev/zero of=/home/RB/temp/extraswap bs=1M count=16384 add 16gb swap file mkswap /home/RB/temp/extraswap swapon /home/RB/temp/extraswap swapon -s and see added swap

The pgsql data file was in /usr/share in the root directory (50gb). That was overdosed also so I moved /usr/share/pgsql to /home/RB/proj/Maps where mucho disk space is available. Good to go after that.

The 2gb canada-latest required nearly 14gb of memory to process so there is a memory leak or poor architecture someplace in the osm2pgsql program -Rick Brown in Reno, NV

permanent link

answered 28 Aug '15, 17:44

RickBrown's gravatar image

RickBrown
111
accept rate: 0%

The github for osm2pgsql is here:

https://github.com/openstreetmap/osm2pgsql/issues

I'd suggest explaining the problem there, and include details of the full command line and version that you used (since what osm2pgsql does depends very much on what you ask it to do - whether you ask it to create an updatable database, for example).

If you can identify a "memory leak or poor architecture" in the code (or even better, suggest a fix) I'd describe that too.

(28 Aug '15, 17:50) SomeoneElse ♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×196

question asked: 05 Oct '11, 18:08

question was seen: 13,466 times

last updated: 28 Aug '15, 17:50

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum