Why not ask your question on the new OpenStreetMap Community Forum?

Greetings again!

For reference, here's my previous thread. Newbie here with this software. https://help.openstreetmap.org/questions/86651/third-time-apologies-if-this-is-a-repeat-nominatim-install-woes

I was doing so great! I thought I was gonna break some kind of speed record, because this was going to complete in like one day (or so I thought). But it didn't - crashed and burned. (crys)

Below is the output...any help? What do I do now?

$ nominatim import --osm-file planet-latest.osm 2>&1 | tee setup.log

2023-02-09 16:09:11: Using project directory: /home/data/1941/nom 2023-02-09 16:09:13: Creating database env.defaults gbpostcodes.csv.gz planet-latest.osm setup.log uspostcodes.csv.gz website wikimedia-importance.sql.gz worldfile

2023-02-09 16:09:14: Setting up country tables 2023-02-09 16:09:15: Importing OSM data file 2023-02-09 16:09:15 osm2pgsql version 1.7.1 2023-02-09 16:09:15 Database version: 12.13 (Ubuntu 12.13-0ubuntu0.20.04.1) 2023-02-09 16:09:15 PostGIS version: 3.0 2023-02-09 16:09:16 Parsing gazetteer style file '/usr/local/etc/nominatim/import-extratags.style'. NOTICE: table "place" does not exist, skipping 2023-02-10 11:05:12 Reading input files done in 68156s (18h 55m 56s).

2023-02-10 11:05:12 Processed 8172948111 nodes in 20393s (5h 39m 53s) - 401k/s 2023-02-10 11:05:12 Processed 915512510 ways in 36157s (10h 2m 37s) - 25k/s

2023-02-10 11:05:12 Processed 10604686 relations in 11606s (3h 13m 26s) - 914/s 2023-02-10 11:05:12 Done postprocessing on table 'planetosmnodes' in 0s 2023-02-10 11:05:12 Done postprocessing on table 'planetosmways' in 0s 2023-02-10 11:05:12 Done postprocessing on table 'planetosmrels' in 0s 2023-02-10 11:05:12 osm2pgsql took 68157s (18h 55m 57s) overall. 2023-02-10 11:05:13: Importing wikipedia importance data 2023-02-10 11:07:11: Importing secondary importance raster data 2023-02-10 11:07:11: Secondary importance file not imported. Falling back to default ranking. 2 023-02-10 11:07:11: Create functions (1st pass) 2023-02-10 11:07:11: Create tables 2023-02-10 11:13:59: Create functions (2nd pass) 2023-02-10 11:14:00: Create table triggers 2023-02-10 11:14:00: Create partition tables 2023-02-10 11:14:01: Create functions (3rd pass) 2023-02-10 11:14:01: Initialise tables 2023-02-10 11:14:01: Load data into placex table

Traceback (most recent call last): File "/usr/local/bin/nominatim", line 14, in <module> exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module', File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim return parser.run(**kwargs) File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run return args.command.run(args) File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 121, in run database_import.load_data(args.config.get_libpq_dsn(), num_threads) File "/usr/local/lib/nominatim/lib-python/nominatim/tools/database_import.py", line 195, in load_data conn = DBConnection(dsn) File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 74, in init self.connect(cursor_factory=cursor_factory) File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 99, in connect self.wait() File "/usr/local/lib/nominatim/lib-python/nominatim/db/async_connection.py", line 128, in wait wait_select(self.conn) File "/home/jpatterson/anaconda3/envs/jpconda/lib/python3.8/site-packages/psycopg2/extras.py", line 758, in wait_select state = conn.poll() psycopg2.OperationalError: FATAL: sorry, too many clients already

asked 10 Feb, 17:37

james000222's gravatar image

james000222
11223
accept rate: 0%

Also my posts do not seem to be formatting correctly. Am I posting things wrong? I'm sorry I don't know why the hard returns are not carrying over.

(10 Feb, 17:38) james000222

Thisforum understands Markdown, so indent e.g. server errror logs by 4 spaces.

Also, when editing xml text what you see in the preview does not match what you get on the page!

(10 Feb, 17:46) SomeoneElse ♦

To clarify, I did the recommended psql tweaks recommended here: https://nominatim.org/release-docs/latest/admin/Installation/

shared_buffers = 2GB maintenance_work_mem = (10GB) autovacuum_work_mem = 2GB work_mem = (50MB) effective_cache_size = (24GB) synchronous_commit = off max_wal_size = 1GB checkpoint_timeout = 10min checkpoint_completion_target = 0.9

But i found a reference here to increase max_connections.

https://support.infoworks.io/support/solutions/articles/14000122840-psycopg2-operationalerror-fatal-sorry-too-many-clients-already

I could try that...but do I just restart the import after that?

permanent link

answered 10 Feb, 17:51

james000222's gravatar image

james000222
11223
accept rate: 0%

SO, I increased this: max_connections=500 Restarted the DB server, and tried this: /home/data/1941/nom$ nominatim add-data --file planet-latest.osm

And it died...

2023-02-10 14:58:08: Using project directory: /home/data/1941/nom 2023-02-10 14:58:08 osm2pgsql version 1.7.1 2023-02-10 14:58:08 Database version: 12.13 (Ubuntu 12.13-0ubuntu0.20.04.1) 2023-02-10 14:58:08 PostGIS version: 3.0 2023-02-10 14:58:09 Parsing gazetteer style file '/usr/local/etc/nominatim/import-extratags.style'. Processing: Node(1210k 1.2k/s) Way(0k 0.00k/s) Relation(0 0.0/s) 2023-02-10 15:15:20 ERROR: DB copy thread failed: Ending COPY mode for 'place' failed: ERROR: relation "place_to_be_deleted" does not exist LINE 1: DELETE FROM place_to_be_deleted pdel ^ QUERY: DELETE FROM place_to_be_deleted pdel WHERE pdel.osm_type = NEW.osm_type and pdel.osm_id = NEW.osm_id and pdel.class = NEW.class CONTEXT: PL/pgSQL function place_insert() line 25 at SQL statement COPY place, line 1: "1 N man_made mast "name"=>"Monte Piselli - San Giacomo" 15 \N "communication:microwave"=>"yes","comm..." . Traceback (most recent call last): File "/usr/local/bin/nominatim", line 14, in <module> exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module', File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim return parser.run(**kwargs) File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run return args.command.run(args) File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/add_data.py", line 79, in run return add_osm_data.add_data_from_file(cast(str, args.file or args.diff), File "/usr/local/lib/nominatim/lib-python/nominatim/tools/add_osm_data.py", line 25, in add_data_from_file run_osm2pgsql(options) File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 159, in run_osm2pgsql subprocess.run(cmd, cwd=options.get('cwd', '.'), File "/home/jpatterson/anaconda3/envs/jpconda/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '1000', '--style', '/usr/local/etc/nominatim/import-extratags.style', '--output', 'gazetteer', '--append', '--flat-nodes', '/home/data/1941/nom/worldfile', '--with-forward-dependencies', 'false', 'planet-latest.osm']' returned non-zero exit status 2.

NO love?

permanent link

answered 11 Feb, 01:12

james000222's gravatar image

james000222
11223
accept rate: 0%

"FATAL: sorry, too many clients already"

Increasing the max_connections was good. The osm2pgsql step of the import runs only 1 processed. Then follows the indexng step where python will create one process per CPU core. That's where the original error occured.

Recovering from that doesn't seem to work. I'd recommend deleting the whole database again and use nominatim import instead of nominatim add-data. The add-data already assumes whatever is in the database finished importing.

When you see lines like Done 40691108 in 14984 @ 2715.599 per second - rank 26 ETA (seconds): 21999.57 That is usually recoverable by running nominatim index. It's really only useful if it crashed near the end and advanced because nominatim import does many steps one would have to run manually. Thus in this case dropping the database and starting with nominatim import is more straight forward.

permanent link

answered 11 Feb, 22:15

mtmail's gravatar image

mtmail
4.7k1574
accept rate: 27%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×685

question asked: 10 Feb, 17:37

question was seen: 500 times

last updated: 11 Feb, 22:15

powered by OSQA