NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Re-posted from stackoverflow:

I'm importing the whole-planet OSM data set into PostgreSQL 9.6.2 on a Amazon EC2 i3.4xlarge (122 Gb mem, 16 CPUs) with the PostgreSQL data directory on a throughput-optimized 2TB volume. I have adjusted Postgres parameters like this (for the import):

shared_buffers = 4GB
work_mem = 100MB
fsync = off 
synchronous_commit = off
max_wal_size = 4GB
checkpoint_completion_target = 0.9

I'm using osm2pgsql to execute the import. Here's the command:

osm2pgsql -c -d gis --number-processes 12 --slim -C 64000 --flat-nodes /data-cache/flat-node-cache/flat.nodes /data-postgres/planet-latest.osm.pbf

The initial steps of the import have gone extremely fast, but its been sitting at Building index on table: planet_osm_ways for days without change. htop indicates active Postgres processeses for checkpoint process and CREATE INDEX. When I log into Postgres and check select * from pg_stat_activity; I see an active state for CREATE INDEX planet_osm_ways_nodes ON planet_osm_ways USING gin (nodes) WITH (FASTUPDATE=OFF) ;

Following the advice in this post I looked for the table place, but the response was Did not find any relation named "place".

Does this sound normal? This is time-sensitive and having already been waiting the better part of a week, I am a little worried something has wrong. Are their other Postgres settings that could speed this part of the process?

asked 01 May '17, 14:35

rgwozdz's gravatar image

rgwozdz
36337
accept rate: 0%


How long did the rest of the import take? IMHO I would wait for the indexing to complete, it is a rather large part of the total time used, and will complete at some point in time.

Note: the reference to a place table is a red herring, that is used for imports in to the Nominatim schema, not the normal osm2pgsql tile rendering schema.

permanent link

answered 01 May '17, 18:31

SimonPoole's gravatar image

SimonPoole ♦
44.7k13326701
accept rate: 18%

edited 01 May '17, 20:23

1

@SimonPoole - the portion of the import prior to the phase it is currently on took about 24 hours.

(01 May '17, 18:37) rgwozdz
1

Eventually finished, took several more days. Failed on next step because max_connections were too low for that many processors. Success after increasing max_connections.

(11 May '17, 14:52) rgwozdz

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×25

question asked: 01 May '17, 14:35

question was seen: 4,110 times

last updated: 11 May '17, 14:52

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum