Hello,

I created an issue at osm2pgsql's GitHub but it's probably me not correctly using the tools, so I'll ask here as well.

I'm setting up a renderd / Mapnik tile server that I want to keep synced with osmosis / osm2pgsql hourly. Everything seems to work (outputs of osmosis & osm2pgsql looks fine, osmosis does download the changes, and osm2pgsql does something to the database). But I have two problems:

  • if I add / edit a feature on OSM iD, my local database does not get the update (openstreetmap.org did get the update)
  • sometimes features are removed (for instance yesterday I lost the USA boundaries polygon, and today France & Spain boundaries are gone as well). They are removed from the DB, it's not a stylesheet problem

What could be wrong?

Here's the complete set of command I'm using:

1°) Database import

osm2pgsql \
    -d osm \
    --create --slim -G --hstore \
    -C 20000 \
    --number-processes 1 \
    --tag-transform-script ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.lua \
    --style ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.style \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    <path/to/planet-osm.pbf>

The process took ~24h, and created a 719GB database (with last week's planet PBF).

2°) Prepare sync (only once)

# Set the location
WORKOSM_DIR=~/osm-sync

# Get the PBF file timestamp
TIMESTAMP=$(osmium fileinfo <path/to/planet-osm.pbf> | grep osmosis_replication_timestamp= | cut -c 35-54)

YEAR=$(echo $TIMESTAMP | cut -c 1-4)
MONTH=$(echo $TIMESTAMP | cut -c 6-7)
DAY=$(echo $TIMESTAMP | cut -c 9-10)
HOUR=$(echo $TIMESTAMP | cut -c 12-13)

# Download the initial state.txt
wget "https://replicate-sequences.osm.mazdermind.de/?Y=$YEAR&m=$MONTH&d=$DAY&H=$HOUR&i=00&s=00&stream=hour" -O $WORKOSM_DIR/state.txt

# Create the config file
rm -f $WORKOSM_DIR/configuration.txt
osmosis --read-replication-interval-init workingDirectory=$WORKOSM_DIR

# Update the config file
sed -i 's!http:!https:!' $WORKOSM_DIR/configuration.txt sed -i 's!minute!hour!' $WORKOSM_DIR/configuration.txt

3°) Continue to sync (this is called every hour with a CRON task)

# Set some variables
WORKOSM_DIR=~/osm-sync
OSM_CARTO_DIR=<path-to-openstreetmap-carto>
EXPIRE_FILE_NAME=expire-list.txt

# Download changes
osmosis \
    --read-replication-interval workingDirectory=$WORKOSM_DIR \
    --simplify-change \
    --write-xml-change $WORKOSM_DIR/tmp/osmChange.xml

# Apply the changes
osm2pgsql \
    --database gis \
    --append --slim --multi-geometry --hstore \
    --cache 3000 \
    --tag-transform-script $OSM_CARTO_DIR/openstreetmap-carto.lua \
    --style $OSM_CARTO_DIR/openstreetmap-carto.style \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    --expire-tiles 10-20 \
    --expire-output $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME \
    --input-reader xml \
    $WORKOSM_DIR/tmp/osmChange.xml

# Expire tiles in [10-15]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=10 \
    --max-zoom=15 \
    --touch-from=10 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

# Delete tiles in [16-20]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=16 \
    --max-zoom=20 \
    --delete-from=16 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

asked 10 Jul, 19:55

Tim%20Autin's gravatar image

Tim Autin
1634
accept rate: 0%

edited 12 Jul, 13:18

If it helps you to compare with one that works, I run https://github.com/SomeoneElseOSM/mod_tile/blob/zoom/openstreetmap-tiles-update-expire every 5 minutes and I haven't seen the problem that you're seeing.

(13 Jul, 15:59) SomeoneElse ♦

Thanks, I already wrote my script based on this one, they are very similar (osmosis + osm2pgsql and then render_expired). It's even worse, my planet_osm_nodes table is now empty -_- ... What did you use to get the PBF file date, to know when to start?

(13 Jul, 18:59) Tim Autin

If you mean "before applying any updates" then it will have been downloaded from Geofabrik. The database load script is actually https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/update_render.sh .

(13 Jul, 19:59) SomeoneElse ♦

I meant how did you get timestamp to feed the openstreetmap-tiles-update-expire script the first time, to init osmosis with --read-replication-interval-init?

(13 Jul, 20:05) Tim Autin

Oh I see thanks. I'm downloading world data from https://planet.openstreetmap.org, where the timestamp is not given. I'm using "osmium fileinfo $1 | grep osmosis_replication_timestamp=" which seems accurate. Appart from that I can't see what could go wrong :/

(13 Jul, 20:56) Tim Autin

Re, the problem is still occuring with your script. I have the same behavior on 3 different servers. I'm using version 0.96.0 of osm2pgsql, what's yours?

(14 Jul, 10:51) Tim Autin

The same - "osm2pgsql version 0.96.0 (64 bit id space)", built locally from git://github.com/openstreetmap/osm2pgsql.git I believe.

(for completeness) the map is here and the update progress is here.

(14 Jul, 12:41) SomeoneElse ♦

Thanks. I re-built osm2pgsql to get the latest updates of the master branch, ran an apt upgrade, and I'm (again) re-creating my planet db to retry. You created your data with line 205 of your update_render.sh, right? Mine is very similar, but I'm using --hstore and --flat-nodes (without this I get an OOME, despite my 64GB of RAM). Do you know if one of these option could cause the bug?

(14 Jul, 12:59) Tim Autin

Yes - https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/update_render.sh#L204 creates the database, and then appends the two .osm files that contain the map legend. My guess is that "--flat-nodes" is causing the effect that you're seeing. Maybe try with a smaller file (that won't run out of memory) without "--flat-nodes"?

(14 Jul, 13:26) SomeoneElse ♦

The problem with a smaller file is that it's harder to see if features have disappeared: it's easy to spot a missing country, less to see a missing restaurant. I'm trying a planet import without --flat-nodes but with --number-processes 1, I never tried that, and it could work.

(14 Jul, 13:45) Tim Autin

Hi Tim, did you solve this while using a more recent osm2pgsql version?

I'm actually seeing the same behavior with osm2pgsql 0.96 and flat-nodes option - at least regarding your second point of (multi-)polygons getting lost while I can't confirm the first one (new nodes not being added). And I can add the oddity of running servers in parallel - same stack - same updates but polygon losts are different between the systems ...

(14 Nov, 01:01) Spiekerooger

Problem is still there for me, I ended up recreating the whole database roughly every 3 monthes, and I redirect my users to the backup server during the process. A bit cubersome, but at least it works.

(yesterday) Tim Autin

Hi Tim,

thanks for answering.

So I'll try to investigate further what's going one there and report back here.

(yesterday) Spiekerooger
showing 5 of 14 show 9 more comments
Be the first one to answer this question!
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×219
×211
×28
×6

question asked: 10 Jul, 19:55

question was seen: 269 times

last updated: yesterday

powered by OSQA