I am trying to install Nominatim based off of a dockerized version of the tool found here: https://github.com/mediagis/nominatim-docker. I was successful with installing dockerised Nominatim with a smaller country locally. And the process ran pretty fast.

Then, I got an EC2 instance with, 64GB memory, 1TB SSD, and 6 core processors to do an install with the entire planet (48GB). The most of the planet file got processed fast, in about 6hours and when processing the relations gave an error and failed. When I ran the planet file based on the same docker image, the object parsing (specially ways) were extremely slow. About 100 times slower than the initial run. This is the error I got when the planet ran faster:

Processing: Node(5640116k 412.0k/s) Way(625947k 118.89k/s) Relation(628790 712.11/s)2019-12-18 22:48:23.270 UTC [210] LOG:  incomplete message from client
2019-12-18 22:48:23.270 UTC [210] CONTEXT:  COPY place, line 233940695
2019-12-18 22:48:23.270 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.270 UTC [207] LOG:  incomplete message from client
2019-12-18 22:48:23.270 UTC [207] CONTEXT:  COPY planet_osm_rels, line 598784
2019-12-18 22:48:23.270 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
Killed
2019-12-18 22:48:23.275 UTC [210] ERROR:  unexpected EOF on client connection with an open transaction
2019-12-18 22:48:23.275 UTC [210] CONTEXT:  COPY place, line 233940695
2019-12-18 22:48:23.275 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.275 UTC [207] ERROR:  unexpected EOF on client connection with an open transaction
2019-12-18 22:48:23.275 UTC [207] CONTEXT:  COPY planet_osm_rels, line 598784
2019-12-18 22:48:23.275 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
2019-12-18 22:48:23.275 UTC [207] LOG:  could not send data to client: Broken pipe
2019-12-18 22:48:23.275 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
2019-12-18 22:48:23.275 UTC [210] LOG:  could not send data to client: Broken pipe
2019-12-18 22:48:23.275 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.275 UTC [207] FATAL:  terminating connection because protocol synchronization was lost
2019-12-18 22:48:23.325 UTC [210] FATAL:  terminating connection because protocol synchronization was lost
ERROR: No Data
string(7) "No Data"

Finally, I just took a pbf of North America (8GB) and tried the install. Object parsing related to North America went relatively fast, but it is still working on rankings and indexes. The install is going on for over 16 hours.

I didn't do any PostgreSQL optimizations before running the install. When the object parsing is slow, I see the container is only using about 1% of the available memory.

In Nominatim install instructions (here: https://nominatim.org/release-docs/latest/admin/Installation/) I see that with less config (32 GB RAM) a planet install is taking only 2 days. What am I doing wrong and how can I optimize the process?

asked 22 Dec '19, 03:34

picmate's gravatar image

picmate
71359
accept rate: 50%


The mediagis docker image is not particularly well suited for large size imports like North America or the planet. Read the original Nominatim installation instructions to get an idea about tuning and here in particular the section on PostgreSQL tuning, using flatnode files and notes on time and memory use during import.

A machine of your size should manage a planet import in 2-3 days but only if you use flatnode files and a tuned postgres configuration.

permanent link

answered 22 Dec '19, 16:39

lonvia's gravatar image

lonvia
5.7k25381
accept rate: 41%

In the Docker file I don't see "--osm2pgsql-cache" set for the setup.php (https://nominatim.org/release-docs/latest/admin/Import-and-Update/#initial-import-of-the-data) That will give osm2pgsql more RAM to work with.

permanent link

answered 22 Dec '19, 04:17

mtmail's gravatar image

mtmail
4.1k1261
accept rate: 30%

Thanks. Can I know whether 20 ways/sec parsing time is what I should expect for way parsing when working with this type of instance configuration?

(22 Dec '19, 13:27) picmate
1

I had around 6000k/s (node), 60k/s (way), 1000/s (relation) on the last planet import.

(22 Dec '19, 17:09) mtmail
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×617
×604
×230

question asked: 22 Dec '19, 03:34

question was seen: 1,060 times

last updated: 22 Dec '19, 17:09

powered by OSQA