Ways processing extremely slowly (0.08k/s) on planet import

Question

I'm doing a full planet import on an Ubuntu machine with 28GB of RAM. I've changed the postgres.conf proportionally to what is recommended on the wiki and kicked off the import process with the following options:

sudo -u postgres ./utils/setup.php --osm-file /datadrive/downloadedfilesforimport/planet-latest.osm.bz2 --all --osm2pgsql-cache 21000 2>&1 | sudo tee setup.log

Nodes import quickly enough, but Ways are moving at less than 1/1000th the speed in terms of k/s. The process is chugging along and using lot's of memory but relatively little CPU compared to when it was importing Nodes. I know that Ways are supposed to take longer than Nodes to import but what I've read in the benchmarks and elsewhere, this doesn't seem right. Here's the current output:

Import
osm2pgsql SVN version 0.89.0-dev (64bit id space)

Using projection SRS 4326 (Latlong)
NOTICE:  table "place" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=21000MB, maxblocks=2688000*8192, allocation method=11
Mid: pgsql, scale=10000000 cache=21000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /datadrive/downloadedfilesforimport/planet-latest.osm.bz2
Using XML parser.
Processing: Node(3046040k 107.4k/s) Way(3566k 0.08k/s) Relation(0 0.00/s)

Ways started processing about 12 hours ago. Any idea what could be going on here? Should I just be patient?

EDIT: being that I'm from the Windows world, the muscle memory around Ctrl+C to copy is strong and as a result has caused me to kill the import. If I need to adjust settings and restart the import, that's no longer a problem.

Answer 1

Would like to make some contribution here on exactly the same problem: my semi-production nominatim installation test setup initially included 2GB of RAM for VM with Ubuntu 16.04. While installing using the guideline I kept all postgresql buffer sizes to their defaults (e.g. did not increase them to 2G/10G/24G as this manual suggests), and also decreased --osm2pgsql-cache setting by 10x, e.g. running:

./utils/setup.php --osm-file /srv/nominatim/Nominatim-3.1.0/data/russia-latest.osm.pbf
--all --osm2pgsql-cache 2800 2>&1 | tee setup.log

However this attempt failed with exactly the same problem as Op described, producing processing speed as follows (approx. values, as I did not keep the 1st attempt logs):

Processing: Node(290006k 73.0k/s) Way(1003k 0.11k/s) Relation(0 0.0/s)

The thread was stuck for hours at this stage with 0,11k/s throughput for Way processing, so I cancelled the import, decided to increase VM RAM from 2GB to 8GB. This lead to success: both import speed doubled and the processing stage was finished, overall import continued to next stages:

Reading in file: /srv/nominatim/Nominatim-3.1.0/data/russia-latest.osm.pbf
Using PBF parser.
Processing: Node(290006k 154.0k/s) Way(25978k 47.15k/s) Relation(744260 515.77/s)  parse time: 3877s
Node stats: total(290006204), max(5597166795) in 1883s
Way stats: total(25978618), max(585757808) in 551s
Relation stats: total(745112), max(8271126) in 1443s

While writing this lines I'm waiting for Load Data stage completion:

2018-05-05 19:09:29 == Loading word list
 count
--------
 885590
(1 row)

 count
-------
 46228
(1 row)

2018-05-05 19:19:29 == Load Data
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Ways processing extremely slowly (0.08k/s) on planet import

Follow this question

Related questions