SOLVED: Slow pre-rendering with generate_tiles_multiprocess.py

Question

Hi

Using Mapnik 2.2, and the same recipy for installing PostGis, Mapnik and osm2pgsql as used by Koles500.

Earlier I imported europe-latest.osm.pbf to database. The import was done in: 155324s, an import time I'm quite happy with considering that I only had 12GB of ram on my computer while importing.

Yesterday I startet to pre-generate tiles from a portion of the europe area as set by this bounding box: http://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox=3.98,57.69,32.23,71.46

Now generate_tiles_multiprocess.py has only generated 275 tiles in 18 hours and 15 minutes. I have set generate_tiles_multiprocess.py to use 4 threads:

NUM_THREADS = 4

as well as specifying this at the end:

minZoom = 1
maxZoom = 11
bbox = (3.98,57.69,32.23,71.46)
render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom, "Norway")

I can see that the four CPU cores has a load usage of approximately 10 - 30%, most of the times it is about 20%. More than 10GB of memory is not in use.

I do not understand why tile pre-rendering is so slow.

But I do remember that I forgot to remove -l from the options when importing data to the database using osm2pgsql: I used this:

osm2pgsql -s -U postgres -C 8000 --number-processes 4 -l -m -d osm -p planet_osm -E 3857 -S "N:\osm2pgsql\default.style"  T:\planet-osm-2018-07-28\europe-latest.osm.pbf

Can that -l option (--latlng) when importing data using osm2pgsql - cause Mapnik to use extra long time to generate the tiles? If that option is the cause - can I fix this without having to import whole pbf file from Europe again? If it's not the cause, what could be the reason for the extremely slow tile generation?

Computer now with a little bit more ram:

The whole DB on an SSD
Tiledir on a 10000 RPM disk (SATA 2).
3.2 Ghz CPU with 4 cores.
20 GB RAM (5 x 4 GB)
Windows 7

Compared to the specifications on the computer Koles500 uses, I simply do not understand the extreme slowness of pre-rendering time. The few tiles that have been pre-rendered are looking perfect.

If someone has tips on how to improve pre-rendering time (using Mapnik 2.2), please let me know. Waiting more than 100 days for the 41 000 tiles to pre-render is just to long.

Accepted Answer

0

Obviously this is difficult to answer without access to the machine, but

some low zoom tiles are slow to process (because the tiles will cover a lot of data, particularly in parts of Europe), you don't mention which style you are using, it depends on that naturally too.
using the -l option (essentially that means you have the original OSM WGS84 coordinates in your DB) implies that mapnik needs to reproject all the data to web-mercator for rendering (at least for the OSM standard style), this will definitely have a performance impact, but would have to be quantified by benchmarking the two variants. In any case you could try using PostGIS to reproject all the geometry objects in the DB, but I suspect that re-importing is going to be less work.

permanent link

answered 05 Aug '18, 08:53

SimonPoole ♦
44.7k●13●326●701
accept rate: 18%

Thanks for answer. I'm using the osm.xml file which comes with mapnik 2.2, only with minor changes to some colors for water and wood. With the current db data, 4 tiles are generated every 22 minutes (approximately). All 4 tiles are generated within a second. Then 22 minutes to next 4 tiles, and so on.

So I imported iceland-latest.osm.pbf into database without the -l option. Used the same stylesheet but with the bbox for iceland (instead of Norway). Then I got 4006 tiles generated in 2 minutes and 40 seconds. Which is good for me.

So pre-rendering obviously works fast with one small country (hopefully because I imported without the -l option).

Currently I'm importing whole europe again - without using the -l option. I'll come back laiter today or tomorrow and tell if the pre-rendering is faster with the new imported db-data.

I have also read about the maximum-extent parameter for the map tag in the xml file. So with the whole europe imported: If I want to pre-render tiles for Iceland only - do I have to set both the extent (in the layer datasource) and the maximum-extent in the map tag, in addition to using bbox in generate_tiles_multiprocess.py?

(06 Aug '18, 10:26) MapViking

Answer 2

Update. It is now working very good. Imported europe-latest.osm.pbf from geofabrik once more, this time without the -l flag.

Tiles are now pre-rendering pretty fast.

This comment has been changed, since I previously reported that it did not work. By some reason the xml file I used when rendering tiles for iceland did not work. But when I tried the xml file for norway, sweden and finland - it all worked very good.

Thanks for input on removing that -l option for osm2pgsql.

SOLVED: Slow pre-rendering with generate_tiles_multiprocess.py

Follow this question

Related questions