Slow tile rendering and stuck render_list

Hi all, my first post here.

I installed a tile server following this https://switch2osm.org/manually-building-a-tile-server-16-04-2-lts/ and managed to get it work ... sort of. I installed leaflet on top of that, and something indeed happens - eventually. When I go to a new location on the map or a new zoom level, I get an empty screen and after 30 seconds (timeout set in apache conf) 404 errors (GET /hot/3/1/3.png HTTP/1.1" 404 503 "http://10.0.0.126/map.html") and so on. (map of the whole world imported).

When I do a refresh later or move elsewhere and come back, the tiles are there. I am not too familiar with any of this but my diagnosis is that rendering of tiles just takes "too long". This is unlikely to be a hardware issue as I commandeered an unused server from a server room rack for this. I have 40 cores, 192GB of memory and 6TB of solid state storage. This is a fresh 16.04 server install with nothing else running on it and everything up to date.

Ok. Rendering of tiles may be slow, and I thought to speed it up by doing a lot of it beforehand as I do not particularly lack resources here. I tried

./render_list -m default -a -z 0 -Z 10 -n 10

but it just hangs. It does not appear to be doing anything. It does not consume any resources whatsoever and it does not cause anything visible in CPU, renderd or postgresql processes. In contrast, when I navigate with my browser to an unvisited area on the map, I can see 6-8 postgresql processes taking 100% of CPU for a while and renderd peaking occasionally as well.

Apparently something is not right, but what exactly and how could I debug this?

EDIT:

This is my renderd.conf:

[renderd]
num_threads=16
tile_dir=/var/lib/mod_tile
stats_file=/var/run/renderd/renderd.stats

[mapnik]
plugins_dir=/usr/lib/mapnik/3.0/input
font_dir=/usr/share/fonts/truetype
font_dir_recurse=1

[ajt]
URI=/hot/
TILEDIR=/var/lib/mod_tile
XML=/home/osm/src/openstreetmap-carto/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20

From syslog I can see successful rendering but the lower the zoom level, the more time it takes. The higher zoom level speeds are completely acceptable, it is just zoom levels 3-5 that seem to take forever:

Feb  5 14:54:01 foo renderd[2544]: DEBUG: DONE TILE ajt 8 144-151 96-103 in 17.569 seconds
Feb  5 14:56:30 foo renderd[2544]: DEBUG: DONE TILE ajt 4 8-15 8-15 in 154.711 seconds
Feb  5 14:57:38 foo renderd[2544]: DEBUG: DONE TILE ajt 6 32-39 32-39 in 55.559 seconds
Feb  5 14:57:40 foo renderd[2544]: DEBUG: DONE TILE ajt 5 16-23 16-23 in 64.475 seconds
Feb  5 14:58:04 foo renderd[2544]: DEBUG: DONE TILE ajt 7 64-71 72-79 in 1.414 seconds
Feb  5 14:58:08 foo renderd[2544]: DEBUG: DONE TILE ajt 8 136-143 144-151 in 2.584 seconds
Feb  5 14:58:09 foo renderd[2544]: DEBUG: DONE TILE ajt 10 568-575 592-599 in 1.164 seconds

When I run render_list, I do get a connection reported like this (the only interesting line, no other RenderBulk requests appear):

renderd[2544]: DEBUG: Got command RenderBulk fd(8) xml(default), z(1), x(0), y(0), mime(image/png), options()

but then nothing else happens. No other log entries, and no CPU or disk activity whatsoever (I have waited for about an hour for something to happen).

Restarting renderd does not produce any interesting log messages (debug messages about fonts mainly). The only "odd" part is this (no function name - not sure if it is ok or not).

Feb  5 15:54:54 foo renderd[22643]: Starting stats thread
Feb  5 15:54:54 foo renderd[22643]: Loading parameterization function for
Feb  5 15:54:54 foo renderd[22643]: message repeated 15 times: [ Loading parameterization function for]
Feb  5 15:54:55 foo renderd[22643]: Updating max_connection parameter for mapnik layers to reflect thread count
Feb  5 15:54:57 foo renderd[22643]: message repeated 15 times: [ Updating max_connection parameter for mapnik layers to reflect thread count]
Feb  5 15:54:57 foo renderd[22643]: Using web mercator projection settings
Feb  5 15:54:59 foo renderd[22643]: message repeated 15 times: [ Using web mercator projection settings]

EDIT2

If I run render_list as

./render_list -m default -a -z 1 -Z 4 -n 10

I sometimes seem to be getting log messages:

Feb  5 16:00:20 foo renderd[22643]: DEBUG: Got command RenderBulk fd(12) xml(default), z(4), x(8), y(8), mime(image/png), options()
Feb  5 16:00:20 foo renderd[22643]: No map for: default

But it does not seem to matter from functional perspective if these appear or not.