nominatim 4.1.0 initial data import failing on full planet

Hi,

I am having nominatim 4.1.0 crash multiple times while trying to do the initial import. Any help would be greatly appreciated.

I am importing into an Azure VM, machine type Standard E8bs v5 (8 vcpus, 64 GiB memory), disk Premium SSD LRS 2048GiB, OS ubuntu 22.04. I have tried importing the latest planet export which is located at https://planet.openstreetmap.org/pbf/planet-221107.osm.pbf, as well as https://planet.openstreetmap.org/pbf/planet-221031.osm.pbf extract to try to rule out data related issues.

I have included the scripts to the full import log as well as the full script to setup Ubuntu at the end of this message.

The import consistently gets to the same number of Nodes and then an error is thrown and I can't continue. I've tried to delete the database and rerun the import. I've also tried to rebuild the machine, and as previously mentioned, also import a different osm extract. The error stack that is thrown is:

Processing: Node(5013370k 550.7k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last):

File "/usr/local/bin/nominatim", line 14, in <module>
exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module', File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
return parser.run(**kwargs)   File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
return args.command.run(args)   File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 92, in run
database_import.import_osm_data(files, File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 108, in import_osm_data
run_osm2pgsql(options)   File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 152, in run_osm2pgsql
subprocess.run(cmd, cwd=options.get('cwd', '.'),   File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--with-forward-dependencies', 'false', '--log-progress', 'true', '--number-processes', '1', '--cache', '94513', '--output', 'gazetteer', '--style', '/usr/local/etc/nominatim/import-extratags.style', '--create', '/var/osm/extract/import.pbf']' died with <Signals.SIGKILL: 9>.

The command I use to do the initial import is

nominatim import --osm2pgsql-cache 0 --osm-file /var/osm/extract/import.pbf --verbose 2>&1 | tee setup.log

The full setup.log

2022-11-12 22:08:52: Using project directory: /srv/nominatim/nominatim-planet 2022-11-12 22:08:54: Creating database 2022-11-12 22:08:54: Setting up country tables SET SET SET SET SET SET SET SET SET CREATE TABLE COPY 23217 CREATE INDEX 2022-11-12 22:08:55: Importing OSM data file 2022-11-12 22:08:55 osm2pgsql version 1.6.0 2022-11-12 22:08:55 Database version: 14.5 (Ubuntu 14.5-0ubuntu0.22.04.1) 2022-11-12 22:08:55 PostGIS version: 3.2 2022-11-12 22:08:55 Parsing gazetteer style file '/usr/local/etc/nominatim/import-extratags.style'. NOTICE: table "place" does not exist, skipping Processing: Node(5013370k 550.7k/s) Way(0k 0.00k/s) Relation(0 0.0/s)Traceback (most recent call last): File "/usr/local/bin/nominatim", line 14, in <module> exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module', File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim return parser.run(**kwargs) File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run return args.command.run(args) File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 92, in run database_import.import_osm_data(files, File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 108, in import_osm_data run_osm2pgsql(options) File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 152, in run_osm2pgsql subprocess.run(cmd, cwd=options.get('cwd', '.'), File "/usr/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--with-forward-dependencies', 'false', '--log-progress', 'true', '--number-processes', '1', '--cache', '94513', '--output', 'gazetteer', '--style', '/usr/local/etc/nominatim/import-extratags.style', '--create', '/var/osm/extract/import.pbf']' died with <signals.sigkill: 9="">.

Ubuntu Setup Steps

sudo mkdir -p /var/osm/extract sudo chmod 777 /var/osm/extract

#planet

wget -P /var/osm/extract https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf mv /var/osm/extract/planet-latest.osm.pbf /var/osm/extract/import.pbf

sudo useradd -d /srv/nominatim -s /bin/bash -m nominatim su nominatim

sudo apt-get update -qq sudo apt install -y php-cgi sudo apt install -y build-essential cmake g++ libboost-dev libboost-system-dev \ libboost-filesystem-dev libexpat1-dev zlib1g-dev \ libbz2-dev libpq-dev \ postgresql-server-dev-14 postgresql-14-postgis-3 \ postgresql-contrib-14 postgresql-14-postgis-3-scripts \ php-cli php-pgsql php-intl libicu-dev python3-dotenv \ python3-psycopg2 python3-psutil python3-jinja2 \ python3-icu python3-datrie

export USERNAME=nominatim export USERHOME=/srv/nominatim

chmod a+x $USERHOME

sudo sed -i 's/#listen_addresses = '\''localhost'\''/listen_addresses = '\''*'\''/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/shared_buffers = 128MB/shared_buffers = 2GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#maintenance_work_mem = 64MB/maintenance_work_mem = 10GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#autovacuum_work_mem = -1/autovacuum_work_mem = 2GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#work_mem = 4MB/work_mem = 50MB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#effective_cache_size = 4GB/effective_cache_size = 24GB/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#synchronous_commit = on/synchronous_commit = off/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#checkpoint_timeout = 5min/checkpoint_timeout = 10min/g' /etc/postgresql/14/main/postgresql.conf sudo sed -i 's/#checkpoint_completion_target = 0.5/checkpoint_completion_target = 0.9/g' /etc/postgresql/14/main/postgresql.conf

echo '# Allow remote connections, the firewall limits external requests' | sudo tee /etc/postgresql/14/main/pg_hba.conf -a echo 'host all all 0.0.0.0/0 md5' | sudo tee /etc/postgresql/14/main/pg_hba.conf -a

sudo systemctl restart postgresql

sudo -u postgres createuser -s $USERNAME sudo -u postgres createuser www-data

cd $USERHOME wget https://nominatim.org/release/Nominatim-4.1.0.tar.bz2 tar xf Nominatim-4.1.0.tar.bz2 cd Nominatim-4.1.0

wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz

mkdir build cd build cmake .. make sudo make install

mkdir $USERHOME/nominatim-project cd $USERHOME/nominatim-project

sudo apt install -y apache2 libapache2-mod-php

sudo tee /etc/apache2/conf-available/nominatim.conf << EOFAPACHECONF <directory "$userhome="" nominatim-project="" website"=""> Options FollowSymLinks MultiViews AddType text/html .php DirectoryIndex search.php Require all granted </directory>

Alias /nominatim $USERHOME/nominatim-project/website EOFAPACHECONF

sudo a2enconf nominatim sudo systemctl restart apache2

mkdir ~/nominatim-planet cd ~/nominatim-planet

export PROJECT_DIR=~/nominatim-planet

sudo mkdir -p /var/osm/cache sudo chmod 777 /var/osm/cache

wget https://www.nominatim.org/data/wikimedia-importance.sql.gz wget https://www.nominatim.org/data/gb_postcodes.csv.gz wget https://www.nominatim.org/data/us_postcodes.csv.gz

cp ~/Nominatim-4.1.0/settings/env.defaults . sudo sed -i 's/NOMINATIM_FLATNODE_FILE=/NOMINATIM_FLATNODE_FILE=\/var\/osm\/cache\/flatnode.file/g' env.defaults

nominatim import --osm2pgsql-cache 0 --osm-file /var/osm/extract/import.pbf --verbose 2>&1 | tee setup.log

Any help is really appreciated!

Thanks in advance