NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hi,

I’ve been attempting to import the planet.bz2 file into my GIS database but the import fails when my VM aborts during the Processing: Way stage of the import. Details:

  • Machine
  • Windows 7
  • 32 GB RAM
  • Intel i7-3740QM 2.70 GHZ processor
  • 236 GB of disk space
  • 64-bit
  • VirtualBox
  • Version 4.3.4
  • Linux 2.6 (64-bit) running CentOS 6.3
  • PostgreSQL 8.4
  • 27 (of 32) GB RAM
  • 620 GB of disk space (external drive)
  • After several aborts I also edited the Postgresql.conf file using these recommendations (slide 8).
  • osm2pgsql Commands
  • osm2pgsql --username ptosm --password --slim -d gis -C 2048 --style /usr/share/osm2pgsql/default.style /pgsql/planet_tile_file.osm.bz2
  • osm2pgsql --username ptosm --password --slim -d gis -C 12000 --style /usr/share/osm2pgsql/default.style /pgsql/planet_tile_file.osm.bz2
  • Log: Not included because the import runs normally until aborting around (estimate) Processing Node: (2,050,000kb; 36.9 kb/s) Processing: Way (650kb; .07kb/s).

Is my machine capable of completing the planet file import? I’ve also asked about this on the VirtualBox forum but I’m asking here in case it’s an OSM-related issue. Any help or advice would be greatly appreciated!

Thanks,

Alpha

asked 06 Dec '13, 18:01

Alpha47's gravatar image

Alpha47
1223
accept rate: 0%

edited 06 Dec '13, 18:02

How fast is access to the external drive? When you ran with extracts did it appear to be IO-bound?

(06 Dec '13, 21:58) SomeoneElse ♦

At the time it didn't seem to be but I may not have noticed because these were smaller files that I was able to import much more rapidly than the planet file. I'll update this ticket when I have more information.

Thanks for the response!

(07 Dec '13, 22:45) Alpha47

At least on paper you should be able to import a full planet (no claims about how fast/slow it will be). It would however seem to be rather unlikely that the machine will actually continue to run if the VM starts using the 32GB of memory.

You are using both an old postgresql version and command line options that might have been appropriate 3 years ago, but are more than outdated today (three years ago the planet file contained less than half the nodes it has today). You will need to specify at least -C 18000 and likely it would make sense to use the flat file node store. See for example switch2osm.org for updated information.

Without further information on any errors displayed during the abort it will be difficult to help you with the specific problem.

In general it is a good idea to import a small extract first to check that everything works before embarking on a full planet import.

permanent link

answered 06 Dec '13, 20:29

SimonPoole's gravatar image

SimonPoole ♦
44.7k13326701
accept rate: 18%

edited 06 Dec '13, 22:23

Thanks for the response!

Smaller files (a world extract and a couple continent files) all worked successfully, which is nice in light of the inferior commands and postgres database I was using.

I'll upgrade and adjust my command line options and update this ticket when I have more information.

(06 Dec '13, 21:01) Alpha47
1

BTW you should be importing the data in PBF format, it is quite a bit faster.

(06 Dec '13, 23:05) SimonPoole ♦

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×710
×56
×37
×7

question asked: 06 Dec '13, 18:01

question was seen: 7,268 times

last updated: 07 Dec '13, 22:45

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum