NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hello,

I just set up a planet DB on my server, following the switch2osm tutorial (https://switch2osm.org/manually-building-a-tile-server-18-04-lts). I'm using the default openstreetmap-carto style. Tiles rendering is way too slow to be usable: I plan to pre-render up to zoom 12, and generating a single metatile at zoom 13 takes 20 seconds. It still takes 6 seconds at zoom 16.

Here's what I've done so far to speed things up:

What more should I do? I added below more information about my setup. Interestingly rendering durations with render_list seems fine (see below).


Here's my server's config:

  • Intel Core i7-8700 Hexa-Core (12 threads)
  • 64 GB of RAM
  • 2 TB SSD (for the DB)
  • 2 * 6 TB HDD (set up in RAID 1, for the OS & the tile cache)

I imported the planet.pbf file using this command:

time osm2pgsql -d gis --create --slim
-G --hstore -C 48000 --number-processes 6 \
     --tag-transform-script ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.lua \
     --style ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.style \
    planet-latest.osm.pbf

It took 36 hours to complete, creating a DB of around 900 GB.

Here's what I added to my postgresql.conf:

synchronous_commit = off
fsync = off
random_page_cost = 1.1
wal_buffers = 16MB
effective_cache_size = 48GB
maintenance_work_mem = 4GB
shared_buffers = 8MB
work_mem = 128MB
autovacuum = on
effective_io_concurrency = 200
max_connections = 300
checkpoint_completion_target = 0.7
default_statistics_target = 100
min_wal_size = 1GB
max_wal_size = 2GB
max_worker_processes = 12
max_parallel_workers_per_gather = 6
max_parallel_workers = 12

I also changed my cpufreq-set with the following command:

sudo cpufreq-set -g performance

Here are some rendering durations I got with render_list. They seem fine to me:

  • zooms 0 -> 4: 13mn
  • zooms 5 -> 6: 15mn
  • zooms 7: 13mn

What more can I do? Are we supposed to create other indexes than the ones listed in the indexes.sql script?

I was expecting rendering performance to be something like 1-5 secs on that server (depending on the zoom level). Is that realistic?

asked 20 Aug '18, 17:15

timautin's gravatar image

timautin
1514512
accept rate: 0%

edited 20 Aug '18, 19:34

Did you add indexes to your database?

https://github.com/gravitystorm/openstreetmap-carto/blob/master/indexes.sql

Low zoom tiles are slow to render anyway because of reading a lot of data, but that's probably too long.

(20 Aug '18, 17:29) kocio

Thanks for your comment. As said in my question yes, I executed the indexes.sql script (it was taking half an hour before doing so). Yes it seems too slow to me. It should take less than a second at zoom 16, right?

(20 Aug '18, 17:30) timautin

Sorry TL;DR...

It's hard to say for me, because my render_speedtest numbers were taken with just one country in the database:

https://github.com/gravitystorm/openstreetmap-carto/issues/1287#issuecomment-383266123

(20 Aug '18, 18:11) kocio

Thanks, I'm running the test. Stuck at zoom 0 at the moment -_- ... It was very fast for a small extract (I did test with Azerbaidjan & Corsica extracts, I was under the second all the time). I'm wondering if everything is indeed on the SSD, maybe some indexes are on the default partition, I need to find a way to check that.

(20 Aug '18, 18:18) timautin

I think that without whole planet loaded it's kind of dry run test, because probably it renders empty places most of the time. But it's just a wild guess. Anyway, even dry run data can be usable in a limited way.

(20 Aug '18, 18:59) kocio

Well it's still stuck at zoom 0, I gave up. But the numbers you got seems good, I'd like to get the same. I moved the whole DB on the SSD (previously I was only using a table space), it didn't change anything.

(20 Aug '18, 19:06) timautin

Might be worth noting that my cores are never at 100% (5-20%), and my RAM usage is always low (10GB out of my 64GB).

(20 Aug '18, 19:18) timautin

I have an oldish import of Europe and I also experience big slowdowns. Importing does not take so long; last time it was a ~16TiB .pfb file that loaded in a couple of days (256TiB consumer grade SSD, 16GiB of RAM, not much tweaking on the PG side).

I do think that any performance test should be done with a big import, otherwise the penalties for continent/world levels aren't really felt. Current planet is at 41GiB, Europe 18GiB, N.America/Asia 7GiB, maybe something at least 1GiB?

(22 Aug '18, 22:43) Marcos Dione
showing 5 of 8 show 3 more comments

I have an oldish import of Europe and I also experience big slowdowns. Importing does not take so long; last time it was a ~16TiB .pfb file that loaded in a couple of days (256TiB consumer grade SSD, 16GiB of RAM, not much tweaking on the PG side).

I do think that any performance test should be done with a big import, otherwise the penalties for continent/world levels aren't really felt. Current planet is at 41GiB, Europe 18GiB, N.America/Asia 7GiB, maybe something at least 1GiB?

permanent link

answered 22 Aug '18, 22:42

Marcos%20Dione's gravatar image

Marcos Dione
71113
accept rate: 0%

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×440
×165
×33

question asked: 20 Aug '18, 17:15

question was seen: 5,815 times

last updated: 22 Aug '18, 22:43

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum