Improving performance of renderd

Question

Hello, I am running my own OSM server setup as per the instructions on switch2osm.org (Manually building a tile server (18.04 LTS).

Renderd (or rather PostgreSQL) is working quite slowly. From everything I have read, this is to be expected the first time that it generates tiles for an area.

However, are there any performance tune-ups that anyone can suggest to speed up PostgreSQL specifically for Renderd?

I have a m5.2xlarge machine in AWS EC2 (32 GB RAM, 1.5TB SSD (EBS), 8 vCPUs (4 Cores x 2 Threads per Core)). I think that this size of machine should perform faster than it is right now.

Here are some sample renderd results from "/var/log/syslog"

Apr  1 07:54:36 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 17 83680-83687 49440-49447, age 4.69 days
Apr  1 07:54:36 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 17 83672-83679 49440-49447, age 4.69 days
Apr  1 07:54:36 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 16 41840-41847 24720-24727, age 4.12 days
Apr  1 07:54:36 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 16 41832-41839 24720-24727, age 4.12 days
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 17 83672-83679 49440-49447 in 3.643 seconds
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 15 20912-20919 12360-12367, age 3.68 days
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 17 83680-83687 49440-49447 in 3.822 seconds
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 15 20920-20927 12360-12367, age 3.56 days
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 16 41840-41847 24720-24727 in 3.339 seconds
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 14 10456-10463 6176-6183, age 3.62 days
Apr  1 07:54:40 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 16 41832-41839 24720-24727 in 3.657 seconds
Apr  1 07:54:43 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 15 20920-20927 12360-12367 in 3.146 seconds
Apr  1 07:54:43 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 15 20912-20919 12360-12367 in 3.624 seconds
Apr  1 07:54:45 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 14 10456-10463 6176-6183 in 5.515 seconds
Apr  1 07:55:13 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 13 5224-5231 3088-3095, age 3.72 days
Apr  1 07:55:18 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 13 5224-5231 3088-3095 in 4.517 seconds
Apr  1 07:55:22 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 10 656-663 384-391, age 3.08 days
Apr  1 07:55:23 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 9 320-327 192-199, age 3.10 days
Apr  1 07:55:23 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 9 328-335 192-199, age 3.10 days
Apr  1 07:55:24 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 8 160-167 88-95, age 3.05 days
Apr  1 07:55:25 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 10 656-663 384-391 in 2.751 seconds
Apr  1 07:55:25 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 8 160-167 96-103, age 3.10 days
Apr  1 07:55:30 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 9 328-335 192-199 in 7.276 seconds
Apr  1 07:55:30 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 7 80-87 48-55, age 3.05 days
Apr  1 07:55:49 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 9 320-327 192-199 in 26.345 seconds
Apr  1 07:55:49 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 7 80-87 40-47, age 3.04 days
Apr  1 07:56:18 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 8 160-167 96-103 in 53.762 seconds
Apr  1 07:56:18 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 8 136-143 88-95, new metatile
Apr  1 07:56:18 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 8 160-167 88-95 in 54.763 seconds
Apr  1 07:56:18 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 7 72-79 48-55, age 3.04 days
Apr  1 07:56:31 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 7 80-87 48-55 in 60.663 seconds
Apr  1 07:56:31 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 6 40-47 16-23, age 3.01 days
Apr  1 08:00:27 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 7 72-79 48-55 in 248.214 seconds
Apr  1 08:00:27 ip-172-30-0-160 renderd[30679]: DEBUG: START TILE ajt 8 136-143 80-87, new metatile
Apr  1 08:39:57 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 7 80-87 40-47 in 2647.823 seconds
Apr  1 09:00:17 ip-172-30-0-160 renderd[30679]: DEBUG: DONE TILE ajt 6 40-47 16-23 in 3826.766 seconds

It gets super slow in that zoom level sweet spot of 6, 7, or 8. I understand that these intermediate zoom levels have the most data to process.

From various places around the web, I have gotten tidbits of information to speedup the performance. - I have set the PostgreSQL "maintenance_work_mem" setting to 64 MB (https://www.youtube.com/watch?v=Lxloo42gl8A) - Renderd "num_threads" is the default 4 (https://wiki.openstreetmap.org/wiki/User:SomeoneElse/Tuning_renderd_memory_usage)

I also came across this StackExchange post (https://gis.stackexchange.com/questions/181728/mod-tile-not-using-all-available-threads) from several years ago that indicates that maybe the right indexes don't exist on my PostgreSQL setup.

I ran the suggested command to see the indexes

SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch 
FROM pg_stat_all_indexes 
WHERE schemaname = 'public' order by 1;

Here is the output

alt text

Can anyone comment on my setup and / or suggest any way of speeding up Renderd/PostgreSQL?

I also tried running render_list to pre-render tiles but it ran for several hours without giving me any indication that it was doing anything - there were no messages on the screen indicating any progress, and "ps -aux" indicated that the total time run by render_list was 0:00. Maybe I was looking in the wrong place for progress information?

Answer 1

1

The numbers from renderd in /var/log/syslog are alright.

You just have to pre_render most of the tiles for at least up to zoom 12.

But the way you are telling it, your prerendering failed. Did you miss the -m {mapname} param or the -a (or --all) in your render_list call? (I think it would be -m ajt in your case).

You should start pre-rendering by calling

render_list -m ajt -z 0 -Z 12 -n 4 -l 20 --all

and watch /var/log/syslog | grep renderd to see if the prerendering is working.

permanent link

answered 02 Apr '20, 18:06

Spiekerooger
3.1k●2●23●56
accept rate: 16%

Thanks for confirming the numbers.

I have run render_list again and am seeing progress. Thanks for pointing out to use "/var/log/syslog | grep renderd". This shows progress being made.

(02 Apr '20, 18:15) Buzz1000

Are you running full planet or just a country/region? If it's the former (planet) than just wait for some days now for the prerendering to run thru.

(02 Apr '20, 18:21) Spiekerooger

Improving performance of renderd

Follow this question

Related questions