I recently imported the nominatim with full planet and queries are undesirable performance. The first query, before the cache is created, takes longer than 20 seconds and sometimes I see this error:
Internal Server Error
Nominatim has encountered an internal error while accessing the database. This may happen because the database is broken or because of a bug in the software. If you think it is a bug, feel free to report it over on Github. Please include the URL that caused the problem and the complete error details below.
Message: Could not lookup place
SQL Error: DB Error: unknown error
Details:
SELECT osm_type, osm_id, class, type, admin_level, rank_search, rank_address, min(place_id) AS place_id, min(parent_place_id) AS parent_place_id, -1 as housenumber, country_code,get_address_by_language(place_id,-1,ARRAY['short_name','name','brand','official_name','ref','type']) AS langaddress, get_name_by_language(name,ARRAY['short_name','name','brand','official_name','ref','type']) AS placename, get_name_by_language(name, ARRAY['ref']) AS ref, avg(ST_X(centroid)) AS lon, avg(ST_Y(centroid)) AS lat, COALESCE(importance,0.75-(rank_search::float/40)) AS importance, (SELECT max(ai_p.importance * (ai_p.rank_address + 2)) FROM place_addressline ai_s, placex ai_p WHERE ai_s.place_id = min(CASE WHEN placex.rank_search < 28 THEN placex.place_id ELSE placex.parent_place_id END) AND ai_p.place_id = ai_s.address_place_id AND ai_s.isaddress AND ai_p.importance is not null) AS addressimportance, (extratags->'place') AS extra_place FROM placex WHERE place_id in (92672355,196842012,4488269,196732908,48310082,160137,91687842,70711723,83690356,77366185,74053027,92460750,82583385,113120861,177719652,86034442,82755806,76866792,83301006,92789547) AND ( placex.rank_address between 0 and 30 OR (extratags->'place') = 'city' ) AND linked_place_id is null GROUP BY osm_type, osm_id, class, type, admin_level, rank_search, rank_address, housenumber, country_code, importance, langaddress, placename, ref, extratags->'place' [nativecode=ERROR: canceling statement due to statement timeout
CONTEXT: PL/pgSQL function get_addressdata(bigint,integer) line 103 at FOR over SELECT rows
PL/pgSQL function get_address_by_language(bigint,integer,text[]) line 12 at FOR over SELECT rows]
Due to the delay of the queries I thought the problem was the lack of some index and then I executed:
nohup ./utils/setup.php --create-search-indices --ignore-errors &
Output:
2018-08-13 20:59:56 == Create Search indices
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
ERROR: relation "idx_word_word_id" already exists
ERROR: relation "idx_search_name_nameaddress_vector" already exists
ERROR: relation "idx_search_name_name_vector" already exists
ERROR: relation "idx_search_name_centroid" already exists
ERROR: relation "idx_place_addressline_address_place_id" already exists
ERROR: relation "idx_placex_rank_address" already exists
ERROR: relation "idx_placex_pendingsector" already exists
ERROR: relation "idx_placex_parent_place_id" already exists
ERROR: relation "idx_placex_reverse_geometry" already exists
ERROR: relation "idx_location_area_country_place_id" already exists
ERROR: relation "idx_osmline_parent_place_id" already exists
ERROR: relation "idx_search_name_country_centroid" already exists
NOTICE: index "place_id_idx" does not exist, skipping
ERROR: relation "idx_place_osm_unique" already exists
ERROR: relation "idx_postcode_id" already exists
ERROR: relation "idx_postcode_postcode" already exists
Summary of warnings:
2018-08-14 02:23:14 == Setup finished.
What does this line mean? NOTICE: index "place_id_idx" does not exist, skipping
After that, I turn on the postgresql log and called the search api with these parameters:
http://nominatim.dev/nominatim/search.php?format=jsonv2&addressdetails=1&q=cancun
Logs:
2018-08-14 14:30:49 UTC [2489-1] www-data@nominatim LOG: duration: 16092.775 ms statement: SELECT osm_type, osm_id, class, type, admin_level, rank_search, rank_address, min(place_id) AS place_id, min(parent_place_id) AS parent_place_id, -1 as housenumber, country_code,get_address_by_language(place_id,-1,ARRAY['short_name:en-en','name:en-en','short_name:en','name:en','short_name','name','brand','official_name:en-en','official_name:en','official_name','ref','type']) AS langaddress, get_name_by_language(name,ARRAY['short_name:en-en','name:en-en','short_name:en','name:en','short_name','name','brand','official_name:en-en','official_name:en','official_name','ref','type']) AS placename, get_name_by_language(name, ARRAY['ref']) AS ref, avg(ST_X(centroid)) AS lon, avg(ST_Y(centroid)) AS lat, COALESCE(importance,0.75-(rank_search::float/40)) AS importance, (SELECT max(ai_p.importance * (ai_p.rank_address + 2)) FROM place_addressline ai_s, placex ai_p WHERE ai_s.place_id = min(CASE WHEN placex.rank_search < 28 THEN placex.place_id ELSE placex.parent_place_id END) AND ai_p.place_id = ai_s.address_place_id AND ai_s.isaddress AND ai_p.importance is not null) AS addressimportance, (extratags->'place') AS extra_place FROM placex WHERE place_id in (197441826,111603036,182595459,139694930,144830394,118512715,74049435,99932703,159450111,141912308,112119380,80303695,80305679,112647794,80522782,115827503,128389572,149585949,74675442,150225528) AND ( placex.rank_address between 0 and 30 OR (extratags->'place') = 'city' ) AND linked_place_id is null GROUP BY osm_type, osm_id, class, type, admin_level, rank_search, rank_address, housenumber, country_code, importance, langaddress, placename, ref, extratags->'place'
2018-08-14 14:31:35 UTC [2490-1] ERROR: function transliteration(text) does not exist at character 23
2018-08-14 14:31:35 UTC [2490-2] HINT: No function matches the given name and argument types. You might need to add explicit type casts.
2018-08-14 14:31:35 UTC [2490-3] QUERY: SELECT gettokenstring(transliteration(name))
2018-08-14 14:31:35 UTC [2490-4] CONTEXT: PL/pgSQL function public.make_standard_name(text) line 5 at assignment
automatic analyze of table "nominatim.public.placex"
Looking at the logs I thought some functions might be missing, and execute this:
nohup ./utils/setup.php --create-functions --ignore-errors &
2018-08-14 14:48:04 == Create Functions
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
NOTICE: drop cascades to function get_addressdata(bigint,integer)
NOTICE: drop cascades to function get_wikipedia_match(hstore,character varying)
Summary of warnings:
2018-08-14 14:48:04 == Setup finished.
Some info about my installation:
Nominatim version 3.1.0
Ec2 t2.2xlarge with T2 Unlimited | vCPU 8 | 32GB RAM | SSD 870 GB gp2 2610/3000 IOPS
Postgresql configurations:
- shared_buffers = 2GB
- maintenance_work_mem = 10GB
- work_mem = 50MB
- effective_cache_size = 24GB
- synchronous_commit = off
- checkpoint_timeout = 10min
- checkpoint_completion_target = 0.9
- fsync = on
- full_page_writes = on
- random_page_cost = 1.1
- wal_buffers = 16MB
- max_worker_processes = 8
df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 9.5M 3.2G 1% /run
/dev/xvda1 844G 674G 170G 80% /
tmpfs 16G 4.0K 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 13M 13M 0 100% /snap/amazon-ssm-agent/295
/dev/loop1 87M 87M 0 100% /snap/core/4917
/dev/loop2 87M 87M 0 100% /snap/core/4830
/dev/loop3 87M 87M 0 100% /snap/core/5145
tmpfs 3.2G 0 3.2G 0% /run/user/1000
free -h
total used free shared buff/cache available
Mem: 31G 171M 29G 115M 1.5G 30G
Swap: 0B 0B 0B
nominatim=# \d search_name
Table "public.search_name"
Column | Type | Modifiers
--------------------+-------------------------+-----------
place_id | bigint |
importance | double precision |
search_rank | smallint |
address_rank | smallint |
name_vector | integer[] |
nameaddress_vector | integer[] |
country_code | character varying(2) |
centroid | geometry(Geometry,4326) |
Indexes:
"idx_search_name_centroid" gist (centroid)
"idx_search_name_name_vector" gin (name_vector) WITH (fastupdate=off)
"idx_search_name_nameaddress_vector" gin (nameaddress_vector) WITH (fastupdate=off)
"idx_search_name_place_id" btree (place_id)
nominatim=# \d placex
Table "public.placex"
Column | Type | Modifiers
-----------------+-----------------------------+-----------
place_id | bigint | not null
parent_place_id | bigint |
linked_place_id | bigint |
importance | double precision |
indexed_date | timestamp without time zone |
geometry_sector | integer |
rank_address | smallint |
rank_search | smallint |
partition | smallint |
indexed_status | smallint |
osm_id | bigint | not null
osm_type | character(1) | not null
class | text | not null
type | text | not null
name | hstore |
admin_level | smallint |
address | hstore |
extratags | hstore |
geometry | geometry(Geometry,4326) | not null
wikipedia | text |
country_code | character varying(2) |
housenumber | text |
postcode | text |
centroid | geometry(Geometry,4326) |
Indexes:
"idx_place_id" UNIQUE, btree (place_id)
"idx_placex_adminname" btree (make_standard_name(name -> 'name'::text), rank_search) WHERE osm_type = 'N'::bpchar AND rank_search < 26
"idx_placex_geometry" gist (geometry)
"idx_placex_linked_place_id" btree (linked_place_id) WHERE linked_place_id IS NOT NULL
"idx_placex_osmid" btree (osm_type, osm_id)
"idx_placex_parent_place_id" btree (parent_place_id) WHERE parent_place_id IS NOT NULL
"idx_placex_pendingsector" btree (rank_search, geometry_sector) WHERE indexed_status > 0
"idx_placex_rank_address" btree (rank_address)
"idx_placex_rank_search" btree (rank_search)
"idx_placex_reverse_geometry" gist (geometry) WHERE rank_search <> 28 AND (name IS NOT NULL OR housenumber IS NOT NULL) AND (class <> ALL (ARRAY['waterway'::text, 'railway'::text, 'tunnel'::text, 'bridge'::text, 'man_made'::text]))
Triggers:
placex_before_delete AFTER DELETE ON placex FOR EACH ROW EXECUTE PROCEDURE placex_delete()
placex_before_insert BEFORE INSERT ON placex FOR EACH ROW EXECUTE PROCEDURE placex_insert()
placex_before_update BEFORE UPDATE ON placex FOR EACH ROW EXECUTE PROCEDURE placex_update()
I did not come up with a solution with no command executed
What else can I do to try to solve this?
Running the ./utils/setup.php --index command seems to stuck in:
Starting rank 4 Done 0 in 0 @ 0.000000 per second - FINISHED