mod_tile/mapnik bottleneck in save_as_png8_hex?

I am setting up a tile server using the latest mod_tile build, so Mapnik 3.0.12. I noticed after doing all optimizations that some tiles were still serving particularly slowly so I:

• Turned all layers off in the XML so it is only rendering the map background
• Changed the tile directory to a RAM-mounted filesystem
• Ran perf for 60 seconds and dragged around a slippy map a whole load to render lots of background-only tiles (not science, I know)

The result is a lot of tile requests with TTFB latencies ranging from 2ms to 600ms and a perf log that looks like this:

A background tile looks like this, no interaction required:

The server has loads of head room (2x Intel Xeon 5500s and 32GB of RAM). Any ideas?

Edit: The tiles with long waits are the ones which cause a new metatile to be rendered. If I watch the requests come through on the server all meta tiles are taking about the same time to render:

renderd[3150]: DEBUG: DONE TILE default 19 2848-2855 424-431 in 0.551 seconds
renderd[3150]: DEBUG: DONE TILE default 19 2848-2855 416-423 in 0.523 seconds
renderd[3174]: DEBUG: DONE TILE default 19 2856-2863 416-423 in 0.557 seconds
renderd[3174]: DEBUG: DONE TILE default 19 2856-2863 416-423 in 0.550 seconds
renderd[3174]: DEBUG: DONE TILE default 19 2864-2871 416-423 in 0.551 seconds
renderd[3174]: DEBUG: DONE TILE default 19 2864-2871 416-423 in 0.561 seconds
renderd[3174]: DEBUG: DONE TILE default 19 2864-2871 416-423 in 0.561 seconds

Is this a regular time for a 2048 x 2048px metatile? If I replace the background image with a colour it drops to about 450ms.