|
Hi Guys, I have successfully build a tile server following below article : https://switch2osm.org/serving-tiles/manually-building-a-tile-server-18-04-lts/ Successfully imported the data for Switzerland into the database and now i would like to keep the latest changes for this map into sync on daily level http://download.geofabrik.de/europe/switzerland-updates/ How i can achieve this is there anywhere step by step guide , or scripts that i can use. |
|
https://switch2osm.org/serving-tiles/updating-as-people-edit/ (it's relatively new there)
permanent link
This answer is marked "community wiki".
That description uses zveriks trim_osc which isn't a good solution in this case. Geofabrik offers local change files already so that is not needed at all.
(12 Feb '21, 15:43)
Jochen Topf
It's going to depend how up to date you want to be. If you're happy with daily updates (and OP here has said they are) then yes you can consume Geofabrik daily diffs direct and not bother trying to trim them down. By all means add an extra sentence to https://github.com/switch2osm/switch2osm.github.io/blob/master/serving-tiles/updating-as-people-edit.md to say that.
(12 Feb '21, 19:00)
SomeoneElse ♦
Thanks guys for the input and the feedback, much appreciated. Yes we're happy if we can achieve to sync the database on daily level, i followed the instructions provided , however when running the script sudo -u renderaccount /home/renderaccount/src/mod_tile/openstreetmap-tiles-update-expire It popup some strange error : rm: cannot remove '/var/lib/mod_tile/dirty_tiles.6877': No such file or directory Command in the script is the below one : Below are the variables used : OSM2PGSQL_BIN=osm2pgsql EXPIRY_MINZOOM=13 EXPIRY_MAXZOOM=20 OSM2PGSQL_OPTIONS="-d $DBNAME --tag-transform-script /home/renderaccount/src/openstreetmap-carto/openstreetmap-carto.lua" BASE_DIR=/var/lib/mod_tile EXPIRY_FILE=$BASE_DIR/dirty_tiles CHANGE_FILE=$BASE_DIR/changes.osc.gz Cant figure it out why it's throwing this error , does the script suppose to download this files from somewhere or simply creates them ?
(13 Feb '21, 08:27)
DavorB0
For completeness, the source of the original script is https://github.com/SomeoneElseOSM/mod_tile/blob/switch2osm/openstreetmap-tiles-update-expire . By default, when you run that script with a date to initialise it, it'll write a configuration to files in /var/lib/mod_tile/.osmosis/ . In there you'll see "configuration.txt". By default obtains minutely diffs https://planet.openstreetmap.org/replication/minute . "rm: cannot remove '/var/lib/mod_tile/dirty_tiles.6877':" just means that something went so far wrong that it didn't produce a list of dirty tiles. Details of the actual osm2pgsql error error are written to a logfile, and where that is is defined at the top of the file at https://github.com/SomeoneElseOSM/mod_tile/blob/switch2osm/openstreetmap-tiles-update-expire#L37 and https://github.com/SomeoneElseOSM/mod_tile/blob/switch2osm/openstreetmap-tiles-update-expire#L46 . If you look in /var/log/tiles/osm2pgsql.log you should see the actual error.
(13 Feb '21, 12:20)
SomeoneElse ♦
That said, as Jochen has noted, if you want to consume daily diffs from Geofabrik, you'll need to make changes to the script or configuration files in /var/lib/mod_tile/.osmosis/ to say where you're getting files from and how much you want to update at once, or as he suggests, do something different.
(13 Feb '21, 12:21)
SomeoneElse ♦
1
Yes indeed if i check the log into the mention file , it show's completely different error : ERROR: DB copy thread failed: Database error: ERROR: column "addr:interpolation" of relation "planet_osm_point" does not exist Guess i will need to figure out a different way so i can manage to apply the daily updates from Geofabrik. I already try to execute with below commands , unfortunately this doesn't work as well. osmosis --read-replication-interval workingDirectory=/home/renderaccount/data --simplify-change --write-xml-change /home/renderaccount/data/changes.osc.gz osm2pgsql --append -d gis -S ~/src/openstreetmap-carto/openstreetmap-carto.style -C 10500 --slim --number-processes 6 /home/renderaccount/data/changes.osc.gz Previously i created the state and configuration file. $ pwd ; ls -l /home/renderaccount/data configuration.txt download.lock state.txt cat configuration.txt -e baseUrl=http://download.geofabrik.de/europe/switzerland-updates maxInterval=3600 cat state.txt original OSM minutely replication sequence number 4407283timestamp=2021-02-08T21\:42\:02Z sequenceNumber=2876
(13 Feb '21, 13:17)
DavorB0
Missing database columns is something that will likely cause a problem however you update the database, and that particular one isn't one I've seen before. Have you perhaps changed or updated the style and not made some needed changes to the database structure? Maybe you need to make sure that "OSM2PGSQL_OPTIONS" matches your database?
(13 Feb '21, 13:27)
SomeoneElse ♦
Yes the variable OSM2PGSQL_OPTIONS matches the database. This is how it is in the script : DBNAME=gis OSM2PGSQL_OPTIONS="-d $DBNAME" I didn't perform any updates on the database or the style. If the database structure somehow it's not good , should i maybe re-deploy new system completely , following the same guide ? and then to try again with the same script to synchronize the changes
(13 Feb '21, 13:43)
DavorB0
How you update data depends on how you loaded it in the first place. As an example, the options that I need to use when updating a different map style (which has a different database format) can be seen at https://github.com/SomeoneElseOSM/mod_tile/blob/zoom/openstreetmap-tiles-update-expire#L25 . I don't know what columns were originally in your database, so can't really comment on that.
(14 Feb '21, 15:40)
SomeoneElse ♦
In order to install and create a tile server i did follow below guide : https://switch2osm.org/serving-tiles/manually-building-a-tile-server-18-04-lts/ Which everything went fine. The load of the data into the Gis database that i used is below one (same one from the guide , only the country is changed ) : osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 10500 --number-processes 4 -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/switzerland-latest.osm.pbf I did not include the '--tag-transform-script' part into the variable OSM2PGSQL_OPTIONS. Suppose this is the reason why the database structure it's changed and now its no good. I will build a new server and will include this parameter before i execute the script once again. Since all of this is new technology for me and i jump from one article to another to understand everything (mod_tile, render, mapnik etc) and reading documentation. Is there an easy way what/where its needed to be changed in the script 'openstreetmap-tiles-update-expire' in order daily updates from Geofabrik for Switzerland to be applied. Thanks for the help.
(14 Feb '21, 18:41)
DavorB0
1
Below article helped me to update the database on daily level : https://wiki.openstreetmap.org/wiki/HowTo_minutely_hstore Unfortunately i am using the whole planet , but the goal to have the changes for Swiss are included as well in the replication file. With the approach from https://switch2osm.org/serving-tiles/updating-as-people-edit/ to update from Geofabrik didn't work for me.
(15 Feb '21, 18:13)
DavorB0
For completeness, this error: can be caused by the database format not matching the default osm2pgsql style file. This didn't use to be a problem, but it appears that following changes at the "OSM Carto" side (actually bugfixes in terms of what database contents are actually needed) it now is one. To resolve it, specify the style file in the OSM2PGSQL_OPTIONS, something like:
(31 Oct '21, 02:44)
SomeoneElse ♦
showing 5 of 12
show 7 more comments
|
