NOTICE: is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum


I created an issue at osm2pgsql's GitHub but it's probably me not correctly using the tools, so I'll ask here as well.

I'm setting up a renderd / Mapnik tile server that I want to keep synced with osmosis / osm2pgsql hourly. Everything seems to work (outputs of osmosis & osm2pgsql looks fine, osmosis does download the changes, and osm2pgsql does something to the database). But I have two problems:

  • if I add / edit a feature on OSM iD, my local database does not get the update ( did get the update)
  • sometimes features are removed (for instance yesterday I lost the USA boundaries polygon, and today France & Spain boundaries are gone as well). They are removed from the DB, it's not a stylesheet problem

What could be wrong?

Here's the complete set of command I'm using:

1°) Database import

osm2pgsql \
    -d osm \
    --create --slim -G --hstore \
    -C 20000 \
    --number-processes 1 \
    --tag-transform-script ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.lua \
    --style ~/src/map-styles/openstreetmap-carto/ \
    --flat-nodes /ssd/flat-nodes/osm.cache \

The process took ~24h, and created a 719GB database (with last week's planet PBF).

2°) Prepare sync (only once)

# Set the location

# Get the PBF file timestamp
TIMESTAMP=$(osmium fileinfo <path/to/planet-osm.pbf> | grep osmosis_replication_timestamp= | cut -c 35-54)

YEAR=$(echo $TIMESTAMP | cut -c 1-4)
MONTH=$(echo $TIMESTAMP | cut -c 6-7)
DAY=$(echo $TIMESTAMP | cut -c 9-10)
HOUR=$(echo $TIMESTAMP | cut -c 12-13)

# Download the initial state.txt
wget "$YEAR&m=$MONTH&d=$DAY&H=$HOUR&i=00&s=00&stream=hour" -O $WORKOSM_DIR/state.txt

# Create the config file
rm -f $WORKOSM_DIR/configuration.txt
osmosis --read-replication-interval-init workingDirectory=$WORKOSM_DIR

# Update the config file
sed -i 's!http:!https:!' $WORKOSM_DIR/configuration.txt sed -i 's!minute!hour!' $WORKOSM_DIR/configuration.txt

3°) Continue to sync (this is called every hour with a CRON task)

# Set some variables

# Download changes
osmosis \
    --read-replication-interval workingDirectory=$WORKOSM_DIR \
    --simplify-change \
    --write-xml-change $WORKOSM_DIR/tmp/osmChange.xml

# Apply the changes
osm2pgsql \
    --database gis \
    --append --slim --multi-geometry --hstore \
    --cache 3000 \
    --tag-transform-script $OSM_CARTO_DIR/openstreetmap-carto.lua \
    --style $OSM_CARTO_DIR/ \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    --expire-tiles 10-20 \
    --expire-output $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME \
    --input-reader xml \

# Expire tiles in [10-15]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=10 \
    --max-zoom=15 \
    --touch-from=10 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

# Delete tiles in [16-20]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=16 \
    --max-zoom=20 \
    --delete-from=16 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

asked 10 Jul '19, 19:55

Tim%20Autin's gravatar image

Tim Autin
accept rate: 0%

edited 12 Jul '19, 13:18

If it helps you to compare with one that works, I run every 5 minutes and I haven't seen the problem that you're seeing.

(13 Jul '19, 15:59) SomeoneElse ♦

Thanks, I already wrote my script based on this one, they are very similar (osmosis + osm2pgsql and then render_expired). It's even worse, my planet_osm_nodes table is now empty -_- ... What did you use to get the PBF file date, to know when to start?

(13 Jul '19, 18:59) Tim Autin

If you mean "before applying any updates" then it will have been downloaded from Geofabrik. The database load script is actually .

(13 Jul '19, 19:59) SomeoneElse ♦

I meant how did you get timestamp to feed the openstreetmap-tiles-update-expire script the first time, to init osmosis with --read-replication-interval-init?

(13 Jul '19, 20:05) Tim Autin

Oh I see thanks. I'm downloading world data from, where the timestamp is not given. I'm using "osmium fileinfo $1 | grep osmosis_replication_timestamp=" which seems accurate. Appart from that I can't see what could go wrong :/

(13 Jul '19, 20:56) Tim Autin

Re, the problem is still occuring with your script. I have the same behavior on 3 different servers. I'm using version 0.96.0 of osm2pgsql, what's yours?

(14 Jul '19, 10:51) Tim Autin

The same - "osm2pgsql version 0.96.0 (64 bit id space)", built locally from git:// I believe.

(for completeness) the map is here and the update progress is here.

(14 Jul '19, 12:41) SomeoneElse ♦

Thanks. I re-built osm2pgsql to get the latest updates of the master branch, ran an apt upgrade, and I'm (again) re-creating my planet db to retry. You created your data with line 205 of your, right? Mine is very similar, but I'm using --hstore and --flat-nodes (without this I get an OOME, despite my 64GB of RAM). Do you know if one of these option could cause the bug?

(14 Jul '19, 12:59) Tim Autin

Yes - creates the database, and then appends the two .osm files that contain the map legend. My guess is that "--flat-nodes" is causing the effect that you're seeing. Maybe try with a smaller file (that won't run out of memory) without "--flat-nodes"?

(14 Jul '19, 13:26) SomeoneElse ♦

The problem with a smaller file is that it's harder to see if features have disappeared: it's easy to spot a missing country, less to see a missing restaurant. I'm trying a planet import without --flat-nodes but with --number-processes 1, I never tried that, and it could work.

(14 Jul '19, 13:45) Tim Autin

Hi Tim, did you solve this while using a more recent osm2pgsql version?

I'm actually seeing the same behavior with osm2pgsql 0.96 and flat-nodes option - at least regarding your second point of (multi-)polygons getting lost while I can't confirm the first one (new nodes not being added). And I can add the oddity of running servers in parallel - same stack - same updates but polygon losts are different between the systems ...

(14 Nov '19, 01:01) Spiekerooger

Problem is still there for me, I ended up recreating the whole database roughly every 3 monthes, and I redirect my users to the backup server during the process. A bit cubersome, but at least it works.

(16 Nov '19, 12:14) Tim Autin

Hi Tim,

thanks for answering.

So I'll try to investigate further what's going one there and report back here.

(16 Nov '19, 15:33) Spiekerooger
showing 5 of 14 show 9 more comments

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text]( "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:


question asked: 10 Jul '19, 19:55

question was seen: 1,747 times

last updated: 16 Nov '19, 15:33

NOTICE: is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum