I have successfully imported europe-latest.osm.pbf in postgresql 10. Now I am using generate_tiles_multiprocess.py (NUM_THREADS=10) to prerender tiles. I have prerendered europe zoom 0-7. That went fine (but that are not that many tiles). Now I am trying to prerender individual countries (zoom 8-17) (Netherlands for example) but even such a small country takes very long. I have it running now for 6 hours and it has generated 6000 tiles which comes down to 3 seconds per tile (30s per tile per core). At this rate Netherlands will take weeks to finish. The specs of my machine are: i9-7900X CPU @ 3.30GHz 3.31 GHz (10 cores) 64 GB RAM 2 TB SSD I have previously rendered a single province of the Netherlands on my 8 year old laptop (with that province being the only data in my postgresql db)(NUM_THREADS=4) and that went fine (effective 1 tile per 0.25s - 1s per tile per core). This leaves me to believe that postgresql is my bottleneck as my current machine is much faster but it has a lot more data to query. I found a lot of information about how to tune postgresql when importing osm data, but little to nothing about best settings for rendering. Are there any settings that make a world of difference? I have the following postgresql settings: - work_mem 16MB - shared_buffers 128MB - effective_cache_size 4GB - maintenance_work_mem 265MB - autovacuum on What other settings are relevant? Which of these should I alter? I also found some information about tuning the stylesheet queries but I am using the commonly used style openstreetmap-carto and was hoping that a greater mind than me had already optimized it. Or are there any specific db indexes that I should create for this particular stylesheet that would speed things up? When I look at top I see 16 postgres processes (not 10) that use between 45% en 65% cpu and use between 0,1% and 0,3% mem. It looks like it is not using much memory: free -m: total used free shared buff/cache available Mem: 49739 6821 1836 528 41082 41791 Swap: 0 0 0 Does anyone know what my bottleneck is? Last and I hope it is not too relevant. I am running Docker containers in VirtualBox (gave it 50GB RAM). p.s. I read [https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy][1] but I already imported without the -l option. [1]: https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspyhttps://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy p.p.s. Here someone also prerenders europe: https://help.openstreetmap.org/questions/54560/slow-generation-of-tiles-for-whole-europe and he claims to generate half a million tiles over a night. That is 100 times faster than what I am seeing. This leaves me to believe that with better tuning I can gain a lot.

I have successfully imported europe-latest.osm.pbf in postgresql 10. Now I am using generate_tiles_multiprocess.py (NUM_THREADS=10) to prerender tiles. I have prerendered europe zoom 0-7. That went fine (but that are not that many tiles). Now I am trying to prerender individual countries (zoom 8-17) (Netherlands for example) but even such a small country takes very long. I have it running now for 6 hours and it has generated 6000 tiles which comes down to 3 seconds per tile. tile (30s per tile per core). At this rate Netherlands will take weeks to finish. The specs of my machine are: i9-7900X CPU @ 3.30GHz 3.31 GHz (10 cores) 64 GB RAM 2 TB SSD I have previously rendered a single province of the Netherlands on my 8 year old laptop (with that province being the only data in my postgresql db) db)(NUM_THREADS=4) and that went fine. fine (effective 1 tile per 0.25s - 1s per tile per core). This leaves me to believe that postgresql is my bottleneck as my current machine is much faster but it has a lot more data to query. I found a lot of information about how to tune postgresql when importing osm data, but little to nothing about best settings for rendering. Are there any settings that make a world of difference? I have the following postgresql settings: - work_mem 16MB - shared_buffers 128MB - effective_cache_size 4GB - maintenance_work_mem 265MB - autovacuum on What other settings are relevant? Which of these should I alter? I also found some information about tuning the stylesheet queries but I am using the commonly used style openstreetmap-carto and was hoping that a greater mind than me had already optimized it. Or are there any specific db indexes that I should create for this particular stylesheet that would speed things up? When I look at top I see 16 postgres processes (not 10) that use between 45% en 65% cpu and use between 0,1% and 0,3% mem. It looks like it is not using much memory: free -m: total used free shared buff/cache available Mem: 49739 6821 1836 528 41082 41791 Swap: 0 0 0 Does anyone know what my bottleneck is? Last and I hope it is not too relevant. I am running Docker containers in VirtualBox (gave it 50GB RAM). p.s. I read [https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy][1] but I already imported without the -l option. [1]: https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy

I have successfully imported europe-latest.osm.pbf in postgresql 10. Now I am using generate_tiles_multiprocess.py (NUM_THREADS=10) to prerender tiles. I have prerendered europe zoom 0-7. That went fine (but that are not that many tiles). Now I am trying to prerender individual countries (zoom 8-17) (Netherlands for example) but even such a small country takes very long. I have it running now for 6 hours and it has generated 6000 tiles which comes down to 3 seconds per tile. At this rate Netherlands will take weeks to finish. The specs of my machine are: i9-7900X CPU @ 3.30GHz 3.31 GHz (10 cores) 64 GB RAM 2 TB SSD I have previously rendered a single province of the Netherlands on my 8 year old laptop (with that province being the only data in my postgresql db) and that went fine. This leaves me to believe that postgresql is my bottleneck as my current machine is much faster but it has a lot more data to query. I found a lot of information about how to tune postgresql when importing osm data, but little to nothing about best settings for rendering. Are there any settings that make a world of difference? I have the following postgresql settings: - work_mem 16MB - shared_buffers 128MB - effective_cache_size 4GB - maintenance_work_mem 265MB - autovacuum on What other settings are relevant? Which of these should I alter? I also found some information about tuning the stylesheet queries but I am using the commonly used style openstreetmap-carto and was hoping that a greater mind than me had already optimized it. Or are there any specific db indexes that I should create for this particular stylesheet that would speed things up? When I look at top I see 23 16 postgres processes (not 10) that use between 45% en 65% cpu and use between 0,1% and 0,3% mem. It looks like it is not using much memory: free -m: total used free shared buff/cache available Mem: 49739 6821 1836 528 41082 41791 Swap: 0 0 0 Does anyone know what my bottleneck is? Last and I hope it is not too relevant. I am running Docker containers in VirtualBox (gave it 50GB RAM). p.s. I read [https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy][1] but I already imported without the -l option. [1]: https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy

I have successfully imported europe-latest.osm.pbf in postgresql 10. Now I am using generate_tiles_multiprocess.py (NUM_THREADS=10) to prerender tiles. I have prerendered europe zoom 0-7. That went fine (but that are not that many tiles). Now I am trying to prerender individual countries (zoom 8-17) (Netherlands for example) but even such a small country takes very long. I have it running now for 6 hours and it has generated 6000 tiles which comes down to 3 seconds per tile. At this rate Netherlands will take weeks to finish. The specs of my machine are: i9-7900X CPU @ 3.30GHz 3.31 GHz (10 cores) 64 GB RAM 2 TB SSD I have previously rendered a single province of the Netherlands on my 8 year old laptop (with that province being the only data in my postgresql db) and that went fine. This leaves me to believe that postgresql is my bottleneck as my current machine is much faster but it has a lot more data to query. I found a lot of information about how to tune postgresql when importing osm data, but little to nothing about best settings for rendering. Are there any settings that make a world of difference? I have the following postgresql settings: - work_mem 16MB - shared_buffers 128MB - effective_cache_size 4GB - maintenance_work_mem 265MB - autovacuum on What other settings are relevant? Which of these should I alter? I also found some information about tuning the stylesheet queries but I am using the commonly used style openstreetmap-carto and was hoping that a greater mind than me had already optimized it. Or are there any specific db indexes that I should create for this particular stylesheet that would speed things up? When I look at top I see 23 postgres processes (not 10) that use between 45% en 65% cpu and use between 0,1% and 0,3% mem. It looks like it is not using much memory: free -m: total used free shared buff/cache available Mem: 49739 6821 1836 528 41082 41791 Swap: 0 0 0 Does anyone know what my bottleneck is? Last and I hope it is not too relevant. I am running Docker containers in VirtualBox (gave it 50GB RAM). p.s. I read [https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy][1] but I already imported without the -l option. [1]: https://help.openstreetmap.org/questions/65115/solved-slow-pre-rendering-with-generate_tiles_multiprocesspy