|
Hello, I'm in the process to set up my own tile server to reduce the load on OSM, provide my own style, and allow bulk downloads. I'd like some help to know what dedicated server I have to choose. My requirements are:
I can afford this setup:
Or this one:
Would they fit? Would they be overkill (I guess they wouldn't, but a man can dream...)? Which would be better? Other question: how much disk space is needed to execute an update (download the planet.osm pbf file, and insert it in the database while still having the map available for users) ? Thanks in advance for your help! |
|
You will need about 800 GB (1 TB is better) for the world-wide database and you can update it regularly with incremental updates. You could also do a full import with the 32 GB RAM is also the absolute minimum for a planet import; at least 48 GB is recommended. Without an SSD, your import will take several weeks and it will be impossible to run updates because one day's worth of updates takes longer than a day to apply. The server you listed initially costs EUR 70/month. For the same price, get a Hetzner EX51-SSD (64 GB RAM, 2x500 GB SSD) with Flexipack and an extra 2 TB of SATA disk for the tile cache, or if you're planning to add more styles and/or have styles with hill shading, you can get a 4 TB or 6 TB disk too. This budget option is not as "solid" as the 1+1 offer, it doesn't have ECC RAM and no mirrored disks (though you can add a second SATA drive for an extra 10 EUR, or use the PX-61 with NVMe and ECC RAM for also 10 EUR more), but the SSD and 64 GB of RAM will really make a big difference. Thanks a lot for that detailed answer! I didn't know about Hetzner, they offer a lot of customisation! Do you have an experience with their support? So you strongly discourage me to go the HDD route, even with 64GB of RAM, even if I'm ok to do monthly updates (or less) and given that the charge is low? I'm a bit scared to go with the no RAID route. I'm not sure how a HDD + SSD RAID1 as you suggest would perform (if I got you correctly?), do you have an experience with that? Also, how much complexity does it adds to have the postgres db on 2 differents SSDs instead of one? Finally, do you have an idea of the size needed for the cache (I plan 1 layer with hill shading, 1 without)?
(27 Jun '18, 15:35)
timautin
Some people do combined SSD+HDD RAIDs but I have no experience with it and would not recommend it. What I recommend is a RAID0 or LVM stripe on the 2x500 GB SSD giving you 1 TB fast storage for the database (looks like one single drive to PostgreSQL), and then storing the operating system and tile cache (and, if required, also hill shading and contour line data) on the additional SATA disk. Having a RAID0 for the data isn't super bad since you can rebuild the database within a day if it should break. RAID1 for the operating system and tile cache is of course a bit safer but hard disks rarely die within their first few years and you'll upgrade the server before that anyway ;) -- I have generally got good service from Hetzner support.
(27 Jun '18, 16:27)
Frederik Ramm ♦
Thanks again! There's another option with Hetzner for 70€: choose the EX51 (normal one with 2*4TB RAID1) and add a 960GB SSD. Seems the best to me (because the HDD will be RAID-ed). I can even also install osm on the HDD and use it in case the SSD dies (and recreate that say once a year. No updates, and it doesn't matter if it takes weeks to do). Last but not least, does 1 TB of space per layer sounds good to you?
(27 Jun '18, 16:39)
timautin
I'm not 100% sure if I maybe mixed up prices with and without VAT, Hetzner is a bit unclear about that sometimes. Double-check before you order ;) your configuration works too. You're unlikely to use more than 500 GB per layer of tile cache unless you pre-render like hell. If you do backup your rendering database to SATA disk, make sure to only backup the tables required for rendering and not those only required for updating.
(27 Jun '18, 16:47)
Frederik Ramm ♦
Oh boy you're right for the VAT! First their website was displaying VAT included prices, and now VAT exclusive. If I open the website in privacy mode, I have VAT included prices again... So that setup is 85€ VAT excluded. Thanks for the cache size, I was expecting more! And thanks for the tip on the backup rendering db. Answer accepted!
(27 Jun '18, 16:58)
timautin
1
Hi Frederick, your comment is very helpful (and the first actually useful I found!). I am in pretty much the same situation of OP, with the need for several styles. If I just render them on the fly, do I need the additional space you were mentioning? Also, would it be viable or completely nuts? Please correct me if I'm wrong! Thanks!
(05 Nov '19, 17:47)
phts
Space requirements have increased a bit since this was posted 18 months ago. If you have multiple styles you will still want to pre-render them at least to about zoom level 12 because doing that on the fly will be too slow. Total tile cache size depends on what your users do, if you have people downloading all tiles for the planet you'll use more! But on an average-use tile server 100 GB per style should be ok for the tile cache. If you need a very large number of styles, check out vector tiles and on-the-fly conversion from vector to raster, but that's a totally different technology from raster tileservers.
(05 Nov '19, 19:34)
Frederik Ramm ♦
1
Thank you Frederick for your response! I checked something like Node.js TileServer GL, it seems to be what I'd need. From the tutorials it seems I'd need mbtiles. Now, I checked on openmaptiles and they aren't free if you have to use them on a commercial website. If I understood correctly should I need to generate mbtiles myself?
(06 Nov '19, 10:35)
phts
showing 5 of 8
show 3 more comments
|
|
You should be fine with 32GB RAM if you use the osm2pgsql "--flat-nodes" option, which puts intermediate nodes data on disk instead of storing in RAM. I recently managed to load the whole of Europe on my 16 GB RAM Core i7 HQ Windows 10 laptop, allocating only 12 GB to an Oracle Virtual Box Ubuntu machine running PostgreSQL and osm2pgsql, and 2x 500GB of virtual disk space for the VM (although the initial Europe import is far smaller than that, I use the additional space for some materialized views). I wrote about it here: https://forum.openstreetmap.org/viewtopic.php?id=62495 Europe is somewhere between 1/3 to 1/2 of planet, so I am now pretty much 100% convinced that the same setup, if using "--flat-nodes", could also happily import planet. Mind you, you do need an SSD like Frederik recommended, mine is a 2TB Samsung EVO drive. The import, with "--drop" option as well since I don't need updates, finished in just 17 hours. As I recommended there as well in the post, keep your osm2pgsql command line as simple as possible, and don't use "-C" cache option or "--number_processes" parameters if you run a limited machine like I did. Leave it to osm2pgsql to figure out its optimal processing settings. |
