Hello,

I created an issue at osm2pgsql's GitHub but it's probably me not correctly using the tools, so I'll ask here as well.

I'm setting up a renderd / Mapnik tile server that I want to keep synced with osmosis / osm2pgsql hourly. Everything seems to work (outputs of osmosis & osm2pgsql looks fine, osmosis does download the changes, and osm2pgsql does something to the database). But I have two problems:

  • if I add / edit a feature on OSM iD, my local database does not get the update (openstreetmap.org did get the update)
  • sometimes features are removed (for instance yesterday I lost the USA boundaries polygon, and today France & Spain boundaries are gone as well). They are removed from the DB, it's not a stylesheet problem

What could be wrong?

Here's the complete set of command I'm using:

1°) Database import

osm2pgsql \
    -d osm \
    --create --slim -G --hstore \
    -C 20000 \
    --number-processes 1 \
    --tag-transform-script ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.lua \
    --style ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.style \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    <path/to/planet-osm.pbf>

The process took ~24h, and created a 719GB database (with last week's planet PBF).

2°) Prepare sync (only once)

# Set the location
WORKOSM_DIR=~/osm-sync

# Get the PBF file timestamp
TIMESTAMP=$(osmium fileinfo <path/to/planet-osm.pbf> | grep osmosis_replication_timestamp= | cut -c 35-54)

YEAR=$(echo $TIMESTAMP | cut -c 1-4)
MONTH=$(echo $TIMESTAMP | cut -c 6-7)
DAY=$(echo $TIMESTAMP | cut -c 9-10)
HOUR=$(echo $TIMESTAMP | cut -c 12-13)

# Download the initial state.txt
wget "https://replicate-sequences.osm.mazdermind.de/?Y=$YEAR&m=$MONTH&d=$DAY&H=$HOUR&i=00&s=00&stream=hour" -O $WORKOSM_DIR/state.txt

# Create the config file
rm -f $WORKOSM_DIR/configuration.txt
osmosis --read-replication-interval-init workingDirectory=$WORKOSM_DIR

# Update the config file
sed -i 's!http:!https:!' $WORKOSM_DIR/configuration.txt sed -i 's!minute!hour!' $WORKOSM_DIR/configuration.txt

3°) Continue to sync (this is called every hour with a CRON task)

# Set some variables
WORKOSM_DIR=~/osm-sync
OSM_CARTO_DIR=<path-to-openstreetmap-carto>
EXPIRE_FILE_NAME=expire-list.txt

# Download changes
osmosis \
    --read-replication-interval workingDirectory=$WORKOSM_DIR \
    --simplify-change \
    --write-xml-change $WORKOSM_DIR/tmp/osmChange.xml

# Apply the changes
osm2pgsql \
    --database gis \
    --append --slim --multi-geometry --hstore \
    --cache 3000 \
    --tag-transform-script $OSM_CARTO_DIR/openstreetmap-carto.lua \
    --style $OSM_CARTO_DIR/openstreetmap-carto.style \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    --expire-tiles 10-20 \
    --expire-output $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME \
    --input-reader xml \
    $WORKOSM_DIR/tmp/osmChange.xml

# Expire tiles in [10-15]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=10 \
    --max-zoom=15 \
    --touch-from=10 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

# Delete tiles in [16-20]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=16 \
    --max-zoom=20 \
    --delete-from=16 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

asked 10 Jul, 19:55

Tim%20Autin's gravatar image

Tim Autin
1612
accept rate: 0%

edited 12 Jul, 13:18

If it helps you to compare with one that works, I run https://github.com/SomeoneElseOSM/mod_tile/blob/zoom/openstreetmap-tiles-update-expire every 5 minutes and I haven't seen the problem that you're seeing.

(13 Jul, 15:59) SomeoneElse ♦

Thanks, I already wrote my script based on this one, they are very similar (osmosis + osm2pgsql and then render_expired). It's even worse, my planet_osm_nodes table is now empty -_- ... What did you use to get the PBF file date, to know when to start?

(13 Jul, 18:59) Tim Autin

If you mean "before applying any updates" then it will have been downloaded from Geofabrik. The database load script is actually https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/update_render.sh .

(13 Jul, 19:59) SomeoneElse ♦

I meant how did you get timestamp to feed the openstreetmap-tiles-update-expire script the first time, to init osmosis with --read-replication-interval-init?

(13 Jul, 20:05) Tim Autin

Oh I see thanks. I'm downloading world data from https://planet.openstreetmap.org, where the timestamp is not given. I'm using "osmium fileinfo $1 | grep osmosis_replication_timestamp=" which seems accurate. Appart from that I can't see what could go wrong :/

(13 Jul, 20:56) Tim Autin

Re, the problem is still occuring with your script. I have the same behavior on 3 different servers. I'm using version 0.96.0 of osm2pgsql, what's yours?

(2 days ago) Tim Autin

The same - "osm2pgsql version 0.96.0 (64 bit id space)", built locally from git://github.com/openstreetmap/osm2pgsql.git I believe.

(for completeness) the map is here and the update progress is here.

(2 days ago) SomeoneElse ♦

Thanks. I re-built osm2pgsql to get the latest updates of the master branch, ran an apt upgrade, and I'm (again) re-creating my planet db to retry. You created your data with line 205 of your update_render.sh, right? Mine is very similar, but I'm using --hstore and --flat-nodes (without this I get an OOME, despite my 64GB of RAM). Do you know if one of these option could cause the bug?

(2 days ago) Tim Autin

Yes - https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/update_render.sh#L204 creates the database, and then appends the two .osm files that contain the map legend. My guess is that "--flat-nodes" is causing the effect that you're seeing. Maybe try with a smaller file (that won't run out of memory) without "--flat-nodes"?

(2 days ago) SomeoneElse ♦

The problem with a smaller file is that it's harder to see if features have disappeared: it's easy to spot a missing country, less to see a missing restaurant. I'm trying a planet import without --flat-nodes but with --number-processes 1, I never tried that, and it could work.

(2 days ago) Tim Autin
showing 5 of 11 show 6 more comments
Be the first one to answer this question!
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×210
×197
×28
×5

question asked: 10 Jul, 19:55

question was seen: 93 times

last updated: 2 days ago

powered by OSQA