Hi,

I am having nominatim 4.1.0 crash multiple times while trying to do the initial import. Any help would be greatly appreciated.

I am importing into an Azure VM, machine type Standard E8bs v5 (8 vcpus, 64 GiB memory), disk Premium SSD LRS 2048GiB, OS ubuntu 22.04. I have tried importing the latest planet export which is located at https://planet.openstreetmap.org/pbf/planet-221107.osm.pbf, as well as https://planet.openstreetmap.org/pbf/planet-221031.osm.pbf extract to try to rule out data related issues.

I have included the scripts to the full import log as well as the full script to setup Ubuntu at the end of this message.

The import consistently gets to the same number of Nodes and then an error is thrown and I can't continue. I've tried to delete the database and rerun the import. I've also tried to rebuild the machine, and as previously mentioned, also import a different osm extract. The error stack that is thrown is:

Processing: Node(5013370k 550.7k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last):

File "/usr/local/bin/nominatim", line 14, in <module>
exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module', File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
return parser.run(**kwargs)   File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
return args.command.run(args)   File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 92, in run
database_import.import_osm_data(files, File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 108, in import_osm_data
run_osm2pgsql(options)   File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 152, in run_osm2pgsql
subprocess.run(cmd, cwd=options.get('cwd', '.'),   File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--with-forward-dependencies', 'false', '--log-progress', 'true', '--number-processes', '1', '--cache', '94513', '--output', 'gazetteer', '--style', '/usr/local/etc/nominatim/import-extratags.style', '--create', '/var/osm/extract/import.pbf']' died with <Signals.SIGKILL: 9>.

The command I use to do the initial import is

nominatim import --osm2pgsql-cache 0 --osm-file /var/osm/extract/import.pbf --verbose 2>&1 | tee setup.log

The full setup.log

2022-11-12 22:08:52: Using project directory: /srv/nominatim/nominatim-planet 2022-11-12 22:08:54: Creating database 2022-11-12 22:08:54: Setting up country tables SET SET SET SET SET SET SET SET SET CREATE TABLE COPY 23217 CREATE INDEX 2022-11-12 22:08:55: Importing OSM data file 2022-11-12 22:08:55 osm2pgsql version 1.6.0 2022-11-12 22:08:55 Database version: 14.5 (Ubuntu 14.5-0ubuntu0.22.04.1) 2022-11-12 22:08:55 PostGIS version: 3.2 2022-11-12 22:08:55 Parsing gazetteer style file '/usr/local/etc/nominatim/import-extratags.style'. NOTICE: table "place" does not exist, skipping Processing: Node(5013370k 550.7k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last): File "/usr/local/bin/nominatim", line 14, in <module> exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module', File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim return parser.run(**kwargs) File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run return args.command.run(args) File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 92, in run database_import.import_osm_data(files, File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 108, in import_osm_data run_osm2pgsql(options) File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 152, in run_osm2pgsql subprocess.run(cmd, cwd=options.get('cwd', '.'), File "/usr/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--with-forward-dependencies', 'false', '--log-progress', 'true', '--number-processes', '1', '--cache', '94513', '--output', 'gazetteer', '--style', '/usr/local/etc/nominatim/import-extratags.style', '--create', '/var/osm/extract/import.pbf']' died with <signals.sigkill: 9="">.

Ubuntu Setup Steps

sudo mkdir -p /var/osm/extract sudo chmod 777 /var/osm/extract

#planet

wget -P /var/osm/extract https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf mv /var/osm/extract/planet-latest.osm.pbf /var/osm/extract/import.pbf

sudo useradd -d /srv/nominatim -s /bin/bash -m nominatim su nominatim

sudo apt-get update -qq sudo apt install -y php-cgi sudo apt install -y build-essential cmake g++ libboost-dev libboost-system-dev \ libboost-filesystem-dev libexpat1-dev zlib1g-dev \ libbz2-dev libpq-dev \ postgresql-server-dev-14 postgresql-14-postgis-3 \ postgresql-contrib-14 postgresql-14-postgis-3-scripts \ php-cli php-pgsql php-intl libicu-dev python3-dotenv \ python3-psycopg2 python3-psutil python3-jinja2 \ python3-icu python3-datrie

export USERNAME=nominatim export USERHOME=/srv/nominatim

chmod a+x $USERHOME

sudo sed -i 's/#listen_addresses = '\''localhost'\''/listen_addresses = '\''*'\''/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/shared_buffers = 128MB/shared_buffers = 2GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#maintenance_work_mem = 64MB/maintenance_work_mem = 10GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#autovacuum_work_mem = -1/autovacuum_work_mem = 2GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#work_mem = 4MB/work_mem = 50MB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#effective_cache_size = 4GB/effective_cache_size = 24GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#synchronous_commit = on/synchronous_commit = off/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#checkpoint_timeout = 5min/checkpoint_timeout = 10min/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#checkpoint_completion_target = 0.5/checkpoint_completion_target = 0.9/g' /etc/postgresql/14/main/postgresql.conf

echo '# Allow remote connections, the firewall limits external requests' | sudo tee /etc/postgresql/14/main/pg_hba.conf -a echo 'host all all 0.0.0.0/0 md5' | sudo tee /etc/postgresql/14/main/pg_hba.conf -a

sudo systemctl restart postgresql

sudo -u postgres createuser -s $USERNAME sudo -u postgres createuser www-data

cd $USERHOME wget https://nominatim.org/release/Nominatim-4.1.0.tar.bz2 tar xf Nominatim-4.1.0.tar.bz2 cd Nominatim-4.1.0

wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz

mkdir build cd build cmake .. make sudo make install

mkdir $USERHOME/nominatim-project cd $USERHOME/nominatim-project

sudo apt install -y apache2 libapache2-mod-php

sudo tee /etc/apache2/conf-available/nominatim.conf << EOFAPACHECONF <directory "$userhome="" nominatim-project="" website"=""> Options FollowSymLinks MultiViews AddType text/html .php DirectoryIndex search.php Require all granted </directory>

Alias /nominatim $USERHOME/nominatim-project/website EOFAPACHECONF

sudo a2enconf nominatim sudo systemctl restart apache2

mkdir ~/nominatim-planet cd ~/nominatim-planet

export PROJECT_DIR=~/nominatim-planet

sudo mkdir -p /var/osm/cache sudo chmod 777 /var/osm/cache

wget https://www.nominatim.org/data/wikimedia-importance.sql.gz wget https://www.nominatim.org/data/gb_postcodes.csv.gz wget https://www.nominatim.org/data/us_postcodes.csv.gz

cp ~/Nominatim-4.1.0/settings/env.defaults . sudo sed -i 's/NOMINATIM_FLATNODE_FILE=/NOMINATIM_FLATNODE_FILE=\/var\/osm\/cache\/flatnode.file/g' env.defaults

nominatim import --osm2pgsql-cache 0 --osm-file /var/osm/extract/import.pbf --verbose 2>&1 | tee setup.log

Any help is really appreciated!

Thanks in advance

asked 13 Nov, 01:57

SuperUniqueName's gravatar image

SuperUniqueName
162
accept rate: 0%


You use the wrong name for your configuration file. The name should be .env. Therefore the flatnode configuration is not picked up by Nominatim and it runs out of memory.

Other things to consider:

  • Don't copy the full default config file. Simply put a .env file in place which contains only the settings you want to change.
  • Don't use the option --osm2pgsql-cache. Nominatim will do the right thing per default. (And apparently not the right thing when --osm2pgsql-cache 0 is given. There is a bug there that needs fixing.)
permanent link

answered 13 Nov, 09:27

lonvia's gravatar image

lonvia
6.1k25686
accept rate: 40%

Hi Ionvia,

Thanks so much for taking the time to read my issue and share your reply. I can see now that the flatfile was not being created, which heavily points to the fact that the configure file was not being read and points to your solution.

I've changed my script from

cp ~/Nominatim-4.1.0/settings/env.defaults . sudo sed -i 's/NOMINATIM_FLATNODE_FILE=/NOMINATIM_FLATNODE_FILE=\/var\/osm\/cache\/flatnode.file/g' env.defaults

to be

echo 'NOMINATIM_FLATNODE_FILE=/var/osm/cache/flatnode.file' | tee .env -a

and have re-run the script.

Not sure where to put improvement ideas for Nominatim but it'd be great if in nominatim import --verbose mode, it output the values of the configuration variables that it's running with e.g. NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim" NOMINATIM_DATABASE_WEBUSER="www-data" ... NOMINATIM_FLATNODE_FILE= or NOMINATIM_FLATNODE_FILE=/var/osm/cache/flatnode.file

as I found it really hard to get visibility over the configuration settings that nominatim was running with.

Alternatively it could had some text to say "Running using configuration file /user/nominatim/nominatim_planet/.env" or "Running with no custom configuration file"

Aside from that, just another huge thanks Ionvia!

Cheers,

(13 Nov, 21:04) SuperUniqueName
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×676
×259
×194
×9

question asked: 13 Nov, 01:57

question was seen: 188 times

last updated: 24 Nov, 07:57

powered by OSQA