We are facing a problem with a closed connection with osm2pgsql when performing the daily update of the database from "download.geofabrik.de", and the region in question is Brazil. We have three environments with the same settings (CPU, memory and disk), but the problem only occurs in one of them, which is stage, dev and pro work well.

Command:

osm2pgsql -a --slim -e13-20 -d dbhom --number-processes 4 --cache 3042 --tag-transform-script /opt/programs/openstreetmap-carto/5.3.1/hom_openstreetmap/openstreetmap-carto/openstreetmap -carto.lua -S /opt/programas/openstreetmap-carto/5.3.1/hom_openstreetmap/openstreetmap-carto/openstreetmap-carto.style -o /opt/appfiles/hom_openstreetmap/dirty_tiles.38216 /opt/appfiles/hom_openstreetmap/changes .osc.gz

Log:

osm2pgsql version 0.92.0 (64 bit id space) ... Reading in file: /opt/appfiles/hom_openstreetmap/changes.osc.gz Using XML parser. Processing: Node(60k 0.2k/s) Way(0k 0.00k/s) Relation(0 0.00/s)node cache: stored: 63588(100.00%), storage efficiency: 48.50% (dense blocks: 2, sparse nodes: 57356), hit rate: -nan% Osm2pgsql failed due to ERROR: stop COPY_END for planet_osm_roads failed: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

On the PostgreSQL side, we have this message below with a few minutes after the connection with the database started. However, the execution in the application part continues and is only finished minutes later.

2022-06-02 09:36:38 -03 [35713]: [1-1] user=usr,db=dbhom LOG: could not receive data from client: Connection reset by peer

We have the application server, where osm2pgsql runs, and we have the database with the same settings both.

  • 4 CPUs
  • 8 GB RAM
  • PostgreSQL 12
  • PostGIS 3.1

The software versions used in the 3 environments are the same and have been recently revised.

asked 02 Jun, 18:20

asuzano's gravatar image

asuzano
26113
accept rate: 0%


I'd start with upgrading your osm2pgsql version to the latest one. You say the environment was "revised", but osm2pgsql 0.92 is from December 2016, and is by now heavily outdated.

If you upgrade to 1.6.0, you'll get the added benefit of the new "flex" configuration options of osm2pgsql (see https://osm2pgsql.org/doc/manual.html and https://osm2pgsql.org/doc/manual.html#the-flex-output), which gives much greater flexibility in defining the output database schema.

Also, 8GB RAM is low end nowadays. Although your Brazil extract may not be that taxing, as being still quite modest in size at 1.4GB PBF, more RAM never hurts for caching purposes on the OS and database level.

permanent link

answered 02 Jun, 20:56

mboeringa's gravatar image

mboeringa
1.5k21527
accept rate: 9%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×258

question asked: 02 Jun, 18:20

question was seen: 379 times

last updated: 02 Jun, 20:56

powered by OSQA