23
10

When looking at the main map on OpenStreetMap.org, some of the data that was added only minutes ago already shows up, whereas other data that was added weeks or months ago isn't visible yet. How does the update process of the main mapnik map work?

asked 14 Jul '10, 14:00

apmon's gravatar image

apmon
4.8k154455
accept rate: 20%

edited 18 Oct '13, 03:14

aseerel4c26's gravatar image

aseerel4c26
13.8k398240


49

The short answer to this question is: The rendering of the main (mapnik) map should typically update within a few (tens of) minutes, however occasionally it can take longer.

The longer answer is a little more complicated, as there are many components that interact in subtle ways to produce the updated results. So I'll try and explain the process, which may be a little technical at points:

First of all, if you edit OpenStreetMap data, your changes get applied to the main database and are immediately visible from the API. The rendering of the map however uses a secondary database that needs to be kept updated from the main database. For this purpose, the main database publishes minutely diffs that are then applied to the rendering database on OpenStreetMap.org and other rendering servers. As the name suggests, these diffs typically are produced once a minute and so the rendering database is typically 1 - 2 minutes behind the main database. Occasionally (although this shouldn't happen) the diffs get stuck though, and this delay may increase a bit. You can see graphs the current delay of imports for the two rendering servers, yevaud and orm.

Once the new data is in the rendering database, it then needs to actually get rendered. However, the server does not have the capacity and resources to rerender all changed tiles immediately, so a clever algorithm tries to only render those tiles that are necessary in an order to try and ensure that the map appears to be as up to date as possible:

When your browser requests a tile from the server, the server checks if it has already rendered an up to date copy of this tile. If it has, then it simply sends back that copy to the client. If the copy the server has on the other hand is found to be out of date, it checks to see if it (the server) is not overloaded (e.g. in peak time) then tries to rerender the tile from the current rendering database and send this back to the client. If however it is overloaded, it returns the out of date copy back to the client. Similarly, if the rendering process takes more than 3 seconds, again the old tile gets returned to the client to not make the browser wait for ages to get the map. In both cases though, the tile gets added to the rendering queue and will typically get rendered in the background in the next hour or so. Therefore if you visit the same area e.g. an hour later, it should then have finished the rendering and you will see the up to date data.

In some cases when the server is severely overloaded, the rendering queue might fill up and then the rendering request may get dropped, which means the area won't get rerendered and updated until someone else goes and views the area when the queues are no longer full, in which case the procedure above applies again.

You can see if the rendering queue is full and thus it is likely that the map currently won't appear to update on these graphs.

Now this leaves the question of how and when does the server decide if a tile is out of date or not: For this purpose, during the import of the diffs into the rendering db, an extra program calculates which tiles need rerendering based on those diffs. As deciding which tiles are effected, is a hard problem without actually rendering everything, this is only based on heuristics. I.e. it is based upon if there are nodes in the tile, or ways that are in the diff have nodes in those tiles. Relations currently get ignored completely in this process. Similarly, for areas, only the perimeter of the area gets expired.

If you suspect that the tile was not correctly expired, there is a way to manually overwrite this expiry mechanism by appending a /dirty to the url which will append the tile to the back of the rendering queue. However, in nearly all cases this should not be necessary.

As tiles only get rerendered once they are viewed, some of the higher zoom levels that are seldomly viewed might therefore appear out of date by quite a bit. It also means that neighbouring tiles that might have been viewed recently are more up to date, and thus show some strange artefacts of different rendering times.

In most cases hitting the refresh button a couple of times a few minutes apart will fix these issues.

On top of all this, is the additional issue of (proxy) caching, both in your browser and server side to reduce load. This may introduce an additional delay of a few hours and may cause the tile not to get rerenderd as the request might never reaches the server to trigger the rerender.

As you see, this whole things is complicated, but in most cases the short answer is sufficient. i.e. it normally updates within a few minutes.

link

answered 14 Jul '10, 14:41

apmon's gravatar image

apmon
4.8k154455
accept rate: 20%

edited 20 Oct '13, 07:07

pnorman's gravatar image

pnorman
1.3k21321

3

apmon excellent explanation, but it would benefit from a small additional mention that coastlines are an exception. Coastlines being updated every so often. (typically twice a month.)

(20 Aug '10, 12:30) Firefishy
1

more information about the coastline process in the wiki: http://wiki.openstreetmap.org/wiki/Coastline#Main_Mapnik_layer

(18 Sep '11, 16:40) dieterdreist
1

Seems like there is heavy daytime load dependency to be seen in the graphs. What about not dropping any tiles from rendering but instead moving them to rendering during night off-peak hours? Finally, higher server load is not a big financial issues, right? I mean in terms of higher electricity comsumption for instance...

(26 Feb '12, 10:14) Kozuch

I can see my changes are being applied to the standard map, but the open cyclemap layer remains unchanged. How often does the open cyclemap get updated? I have been making corrections to some cycle paths that are not connected correctly thus causing routing errors.

(02 May '12, 11:24) paulcooke

The cycle map layer uses Andy Allen's opencyclemap tiles. He says that "The map is updated every week (on a Thursday or Friday, normally)".

(02 May '12, 12:23) SomeoneElse

Does the above described algorithm only apply for zoomlevels >=13 ? I usually always have to manually /dirty tiles with zoomlevel 12, 11 and 10 to have tertiary roads show up on those tiles...

(23 Jan, 12:58) katpatuka
showing 5 of 6 show 1 more comments
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×244
×199
×64
×2

Asked: 14 Jul '10, 14:00

Seen: 11,000 times

Last updated: 23 Jan, 12:58

powered by OSQA