NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

I'm doing a full planet import on an Ubuntu machine with 28GB of RAM. I've changed the postgres.conf proportionally to what is recommended on the wiki and kicked off the import process with the following options:

sudo -u postgres ./utils/setup.php --osm-file /datadrive/downloadedfilesforimport/planet-latest.osm.bz2 --all --osm2pgsql-cache 21000 2>&1 | sudo tee setup.log

Nodes import quickly enough, but Ways are moving at less than 1/1000th the speed in terms of k/s. The process is chugging along and using lot's of memory but relatively little CPU compared to when it was importing Nodes. I know that Ways are supposed to take longer than Nodes to import but what I've read in the benchmarks and elsewhere, this doesn't seem right. Here's the current output:

Import
osm2pgsql SVN version 0.89.0-dev (64bit id space)

Using projection SRS 4326 (Latlong)
NOTICE:  table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=21000MB, maxblocks=2688000*8192, allocation method=11
Mid: pgsql, scale=10000000 cache=21000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /datadrive/downloadedfilesforimport/planet-latest.osm.bz2
Using XML parser.
Processing: Node(3046040k 107.4k/s) Way(3566k 0.08k/s) Relation(0 0.00/s)

Ways started processing about 12 hours ago. Any idea what could be going on here? Should I just be patient?

EDIT: being that I'm from the Windows world, the muscle memory around Ctrl+C to copy is strong and as a result has caused me to kill the import. If I need to adjust settings and restart the import, that's no longer a problem.

asked 07 Oct '15, 14:15

joelmdev's gravatar image

joelmdev
11113
accept rate: 0%

edited 07 Oct '15, 16:46

Have you checked if/how much your system is paging? (In linux getting it wrong terms "swaping") 28GB is likely not really enough memory for a current planet all things considered.

(07 Oct '15, 15:33) SimonPoole ♦

@SimonPoole swap is not currently configured on the machine. It's a fresh VM, and I tried to use proportional RAM settings for postgres and osm2pgsql as what were outlined in the wiki. The wiki had also pointed out that swapping should be avoided, so I didn't pursue it any further. I'm assuming I could create a swap file via the instructions here https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04 but I'd need some guidance on how large to make it and how to tweak the kernel. I can also bump the machine up to 56GB during the import if needed. Also worth noting that I intend to drop the resources on the machine significantly after the import is finished.

(07 Oct '15, 16:26) joelmdev
(07 Oct '15, 16:54) escada
1

Bumping up the memory would likely be a good idea in any case.

(08 Oct '15, 09:48) SimonPoole ♦

I'm sure that more memory would be good, but would memory alone be enough to cause the Ways to import this slowly?

(09 Oct '15, 15:35) joelmdev

Would like to make some contribution here on exactly the same problem: my semi-production nominatim installation test setup initially included 2GB of RAM for VM with Ubuntu 16.04. While installing using the guideline I kept all postgresql buffer sizes to their defaults (e.g. did not increase them to 2G/10G/24G as this manual suggests), and also decreased --osm2pgsql-cache setting by 10x, e.g. running:

./utils/setup.php --osm-file /srv/nominatim/Nominatim-3.1.0/data/russia-latest.osm.pbf
--all --osm2pgsql-cache 2800 2>&1 | tee setup.log

However this attempt failed with exactly the same problem as Op described, producing processing speed as follows (approx. values, as I did not keep the 1st attempt logs):

Processing: Node(290006k 73.0k/s) Way(1003k 0.11k/s) Relation(0 0.0/s)

The thread was stuck for hours at this stage with 0,11k/s throughput for Way processing, so I cancelled the import, decided to increase VM RAM from 2GB to 8GB. This lead to success: both import speed doubled and the processing stage was finished, overall import continued to next stages:

Reading in file: /srv/nominatim/Nominatim-3.1.0/data/russia-latest.osm.pbf
Using PBF parser.
Processing: Node(290006k 154.0k/s) Way(25978k 47.15k/s) Relation(744260 515.77/s)  parse time: 3877s
Node stats: total(290006204), max(5597166795) in 1883s
Way stats: total(25978618), max(585757808) in 551s
Relation stats: total(745112), max(8271126) in 1443s

While writing this lines I'm waiting for Load Data stage completion:

2018-05-05 19:09:29 == Loading word list
 count
--------
 885590
(1 row)

 count
-------
 46228
(1 row)

2018-05-05 19:19:29 == Load Data
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
permanent link

answered 05 May '18, 18:19

Ivan%20M3's gravatar image

Ivan M3
111
accept rate: 0%

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×25
×3

question asked: 07 Oct '15, 14:15

question was seen: 6,885 times

last updated: 05 May '18, 18:19

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum