Local Nominatim install lookups failing

This is my second shot at posting - first went away when I validated my email.

I'm trying to get a local instance of nominatim 3.1.0 running on FreeBSD 11.1 and PG 9.5. I followed the instructions published here: https://nominatim.org/release-docs/latest/admin/Installation/. I did not encounter any errors during the install and the apache/php side of things seems to be working nicely. I installed no additional components (no TIGER, wikipedia, etc.). I installed New Jersey from here: http://download.geofabrik.de/north-america.html

However I can only query by zipcode. Entering a street name, city, town or place name gives no results. Zipcodes return a result, but clicking on the "details" link gives me an error. This example is zip "07960" and the URL that produces the below error is http://myinstall/details.php?place_id=430312

Bad Request 
Nominatim has encountered an error with your request. Details:
Unknown place id.

Poking around in the database, some seemingly important tables are empty:

nominatim=# select count(*) from location_area;
 count
-------
     0
(1 row)

nominatim=# select count(*) from search_name;
 count
-------
     0
(1 row)

Places seem populated:

nominatim=# select count(*) from place;
 count
--------
 329248
(1 row)

nominatim=# select count(*) from placex;
 count
--------
 325948
(1 row)

I'm stumped. I went through the install a second time to make sure I was doing things correctly and to gather logs of the whole process. The gist below contains the cmake output, (g)make output, setup.php output and the pgsql log. Nothing stands out as an error in the setup log, but this is my first try at anything GIS-related so I have no idea what I'm looking at.

https://gist.github.com/sporkman/969be026d7048eda54d1060312f8fedb

Any ideas? The empty tables seem like a big hint, not sure what populates them though.

nominatim local

asked 24 Feb '18, 09:16

spork123
11●1●1●3
accept rate: 0%

Could you please add the contents of your local settings file(build/settings/local.php)?

(24 Feb '18, 12:03) lonvia

Yes, both installation and output of the setup.php look fine. And placex being filled is a good sign. The 'Create Search indices' step taking only 3 second could be suspicious. Best next good step is to try the installation with a small full country, e.g. Luxembourg or Monaco, so we can rule out the input file new-jersey causes this.

(24 Feb '18, 13:35) mtmail

OK, here's more info... local overrides are minimal:

[spork@nominatim /home/nominatim/Nominatim-3.1.0]$ cat build/settings/local.php
<?php

 @define('CONST_Database_Web_User', 'www');
 @define('CONST_Website_BaseURL', '/');

?>

Also php version just in case there's an issue there:

[nominatim@nominatim /usr/home/nominatim/Nominatim-3.1.0]$ php -v
PHP 5.6.32 (cli) (built: Jan  2 2018 01:29:52)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies

Postgresql/related Versions:

postgis24-2.4.2                Geographic objects support for PostgreSQL databases
postgresql95-client-9.5.10     PostgreSQL database (client)
postgresql95-contrib-9.5.10    The contrib utilities from the PostgreSQL distribution
postgresql95-server-9.5.10     PostgreSQL is the most advanced open-source database available anywhere

Also, for reference, the entire nominatim source and build directories are owned by a "nominatim" user, and this user was also used to perform all database operations. I manually "dropdb nominatim" between tests.

I grabbed Luxembourg and verified the checksum is OK:

[nominatim@nominatim /usr/home/nominatim/Nominatim-3.1.0]$ fetch http://download.geofabrik.de/europe/luxembourg-latest.osm.pbf
luxembourg-latest.osm.pbf                     100% of   22 MB  728 kBps 00m32s
[nominatim@nominatim /usr/home/nominatim/Nominatim-3.1.0]$ md5 luxembourg-latest.osm.pbf
MD5 (luxembourg-latest.osm.pbf) = 4042514b653f6732342ae4bc0a4763ad

I searched for a number of towns listed here (https://en.wikipedia.org/wiki/List_of_towns_in_Luxembourg) and came up with no results. I searched on a postcode and after trying a few got a hit on "8070". And like the New Jersey data, the details link lead to the same error (partial URL: details.php?place_id=210598). Like NJ, "location_area" is empty, "place" and "placex" are populated. Not sure what other tables are important. Of note, a dump of this db, uncompressed is about 304MB. There's something in there. :)

I'm attaching the debug output for the successful and unsuccessful searches as screenshots. The setup.php log is here: https://gist.github.com/sporkman/6829ef07576678c884119bb2ee4f3c01

Scanning the log, things that look weird to me:

Line 11 NOTICE: table "place" does not exist, skipping (very early in the process)
As you noted, lots of things are taking just seconds to complete. That said this is a VM with SSD storage and lots of CPU.
Lines 55-80 note that a function and a bunch of tables don't exist, but I assume they get created
All the "indexing ranks" lines state that nothing was done ("Done 0 in 0 @ 0.000000 per second - FINISHED") - this "Done 0" repeats on lines 109-214, so whatever that indexing is supposed to be doing, it doesn't seem to be actually doing any work
Is there a way to get verbose output on what queries are being run here?

(24 Feb '18, 20:04) spork123

ps - sorry for "answering", this is too big for a comment and there's no formatting in comments.

(24 Feb '18, 20:05) spork123

300MB is still reasonable. The country_grid.sql.gz is already 100MB uncompressed and imported for partitioning data regardless. The "NOTICE: table "place" does not exist, skipping" are fine, postgresql doesn't have a silent "create this table if it doesnt exist yet" afaik so this goes to the output. Have you applied any of the http://www.nominatim.org/release-docs/latest/admin/Installation/#postgresql-tuning changes? For such a small import tuning makes no difference, I'm just wondering if there were any Postgresql config changes. Lastly can we rule out the harddrive is full (though usually Postgresql will print an error). I think it's the first time we see this kind of error, or rather lack of error message while looping through 0 places.

(24 Feb '18, 21:22) mtmail

meta @spork123: there is formatting in comments - just no preview. You can use the preview of the answer or use https://daringfireball.net/projects/markdown/dingus for previewing which is more accurate anyway. (note: I have converted your "answer" to a "comment".)

(24 Feb '18, 22:18) aseerel4c26 ♦

I think I lost a reply, but the short version just that drive space is fine.

PG changes: shared_buffers = 1500MB, work_mem = 50MB, maintenance_work_mem = 512MB, synchronous_commit = off (for import), checkpoint_timeout = 10min, checkpoint_completion_target = 0.9 and the rest is stock.

PHP modules/PEAR: json, openssl, pgsql, xml, zlib and pear-1.10.5, pear-DB-1.9.2

I've been stepping through setup.php to see what it does, and now I see that the indexing step is handled by an executable: build/nominatim/nominatim - I'm going to assume that's where the issue is. Any thoughts on debugging that? It's not doing an instant segfault or anything, so that's good. But does it make any interesting assumptions about what OS it's running on?

(24 Feb '18, 23:24) spork123

btw, thanks @aseerel4c26, I had no idea.

So this indexer. Either it's silently failing or it's not seeing whatever input data it wants. Can anyone elaborate on what it's looking for so I can verify the input data exists?

Manual run:

[nominatim@nominatim /usr/home/nominatim/Nominatim-3.1.0/build]$ nominatim/nominatim -i -d nominatim -P 5432 -t  1 -R 4
nominatim version 3.1.0

Starting indexing rank (0 to 4) using 1 threads
Starting rank 0
  Done 0 in 0 @ 0.000000 per second - FINISHED

Starting rank 1
  Done 0 in 0 @ 0.000000 per second - FINISHED
(etc.)

(24 Feb '18, 23:38) spork123

Even more data. Installed using the same process on Ubuntu 16.04 LTS. Same dataset and all, biggest difference was that I saw the index process actually generate quite a bit of output. And it worked as well.

Experiments to narrow this down to the index process that calls the "nominatim" binary:

Copy "broken"/not properly indexed database dump to working ubuntu server, load it up, use "setup.php --create-functions" to create missing functions and note that the previously working Ubuntu setup stopped working with the same results as the FreeBSD install I've been working on
Manually run "setup.php --index" on this "broken" data on the Ubuntu server, and after that process completes, everything is working normally again
Dump this "good" database, copy it to FreeBSD server. As above, manually create functions. This install now works normally.

What does this show? Given a properly indexed db, the FreeBSD version works, which means the frontend is OK, osm2pgsql works, the nominatim.so module for PG works. It also shows that the data the indexer needs to function is present (proven when copying "bad" db to ubuntu and indexing it). So something is up with the indexer, no?

(25 Feb '18, 05:45) spork123

Hey all, is this better handled in a github ticket?

(28 Feb '18, 01:00) spork123

This looks like a compatibility bug in nominatim.index,c and will require some debugging on your side. Please open a new bug on github.

(02 Mar '18, 09:38) lonvia

@spork123: please link the issue here once you have created it.

(02 Mar '18, 19:53) aseerel4c26 ♦

showing 5 of 12 show 7 more comments

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

*italic* or _italic_
**bold** or __bold__
link:[text](http://url.com/ "title")
image?![alt text](/path/img.jpg "title")
numbered list: 1. Foo 2. Bar
to add a line break simply add two spaces to where you would like the new line to be.
basic HTML tags are also supported

learn more about Markdown

Question tags:

nominatim ×689
local ×39

question asked: 24 Feb '18, 09:16

question was seen: 3,039 times

last updated: 02 Mar '18, 19:53

Local Nominatim install lookups failing

Follow this question

Related questions