Planet File GIS Import Fails in Processing: Way

Hi,

I’ve been attempting to import the planet.bz2 file into my GIS database but the import fails when my VM aborts during the Processing: Way stage of the import. Details:

Machine
Windows 7
32 GB RAM
Intel i7-3740QM 2.70 GHZ processor
236 GB of disk space
64-bit
VirtualBox
Version 4.3.4
Linux 2.6 (64-bit) running CentOS 6.3
PostgreSQL 8.4
27 (of 32) GB RAM
620 GB of disk space (external drive)
After several aborts I also edited the Postgresql.conf file using these recommendations (slide 8).
osm2pgsql Commands
osm2pgsql --username ptosm --password --slim -d gis -C 2048 --style /usr/share/osm2pgsql/default.style /pgsql/planet_tile_file.osm.bz2
osm2pgsql --username ptosm --password --slim -d gis -C 12000 --style /usr/share/osm2pgsql/default.style /pgsql/planet_tile_file.osm.bz2
Log: Not included because the import runs normally until aborting around (estimate) Processing Node: (2,050,000kb; 36.9 kb/s) Processing: Way (650kb; .07kb/s).

Is my machine capable of completing the planet file import? I’ve also asked about this on the VirtualBox forum but I’m asking here in case it’s an OSM-related issue. Any help or advice would be greatly appreciated!

Thanks,

Alpha

asked 06 Dec '13, 18:01

Alpha47
1●2●2●3
accept rate: 0%

edited 06 Dec '13, 18:02

How fast is access to the external drive? When you ran with extracts did it appear to be IO-bound?

(06 Dec '13, 21:58) SomeoneElse ♦

At the time it didn't seem to be but I may not have noticed because these were smaller files that I was able to import much more rapidly than the planet file. I'll update this ticket when I have more information.

Thanks for the response!

(07 Dec '13, 22:45) Alpha47

One Answer:

active answers oldest answers newest answers popular answers

At least on paper you should be able to import a full planet (no claims about how fast/slow it will be). It would however seem to be rather unlikely that the machine will actually continue to run if the VM starts using the 32GB of memory.

You are using both an old postgresql version and command line options that might have been appropriate 3 years ago, but are more than outdated today (three years ago the planet file contained less than half the nodes it has today). You will need to specify at least -C 18000 and likely it would make sense to use the flat file node store. See for example switch2osm.org for updated information.

Without further information on any errors displayed during the abort it will be difficult to help you with the specific problem.

In general it is a good idea to import a small extract first to check that everything works before embarking on a full planet import.

permanent link

answered 06 Dec '13, 20:29

SimonPoole ♦
44.7k●13●326●701
accept rate: 18%

edited 06 Dec '13, 22:23

Thanks for the response!

Smaller files (a world extract and a couple continent files) all worked successfully, which is nice in light of the inferior commands and postgres database I was using.

I'll upgrade and adjust my command line options and update this ticket when I have more information.

(06 Dec '13, 21:01) Alpha47

BTW you should be importing the data in PBF format, it is quite a bit faster.

(06 Dec '13, 23:05) SimonPoole ♦

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

*italic* or _italic_
**bold** or __bold__
link:[text](http://url.com/ "title")
image?![alt text](/path/img.jpg "title")
numbered list: 1. Foo 2. Bar
to add a line break simply add two spaces to where you would like the new line to be.
basic HTML tags are also supported

learn more about Markdown

Question tags:

osm ×710
planet ×56
gis ×37
failure ×7

question asked: 06 Dec '13, 18:01

question was seen: 7,292 times

last updated: 07 Dec '13, 22:45

Planet File GIS Import Fails in Processing: Way

Follow this question

Related questions