Full Planet Import Shortest Time

Considering I can spin up any EC2 Instance (even the strongest ones) and tweaking the configuration for maximum import speed, What's the shortest time for a full planet import using osm2pgsql tool?

is there a bash script that fires osm2pgsql with dynamic values based on the current machine that is running it?

planet import

asked 16 Sep '19, 14:14

LiorM
31●5●5●8
accept rate: 0%

One Answer:

active answers oldest answers newest answers popular answers

On a 64 GB RAM machine with fast NVMe disks you will typically achieve a full planet import (with slim mode and flatnodes) in about 14 hours. This is for old-fashioned "real" hardware. In the cloud, if RAM is not an issue at all and if you do not need updates, then the fastest imports are probably those run without --slim. My guess (based on smaller extracts) is that this could cut import times down to about 6 hours, but you'll need something like 512 GB of RAM (and still need fast disks). Using --slim --drop on a lesser-RAM machine will also be faster than 14 hours at the expense of updatability. If you have tons of RAM and need updates, then importing to a RAM disk database might be worth a try.

I am not aware of a script that auto-adapts values.

permanent link

answered 16 Sep '19, 14:40

Frederik Ramm ♦
82.5k●92●720●1273
accept rate: 23%

Can you share the osm2pgsql config so it will run in ~6 hours? Plus - whats the CPU / RAM prerequisites to achieve that?

No need for updates.

(16 Sep '19, 14:49) LiorM

As I said, the 6 hours is a guess of mine; because of the high memory requirements I haven't run osm2pgsql on a planet file without --slim recently. I think if you have 512 GB of RAM and 4-8 cores you should be fine. The exact command line arguments depend on which map style you'll be using etc. but it would likely be something like:

osm2pgsql -d gis --hstore planet-latest.osm.pbf

some styles will expect you to use a special style file, a LUA transforms file, and perhaps recommend the -G option for building proper multipolygons, all of which have the potential of slowing things down a little. If you need LUA transforms, then building osm2gpsql with "luajit" can recover some of the time used for LUA processing. I'm not 100% sure about the --cache parameter, it used to be irrelevant for non-slim mode but you'll have to see. Do report back your results!

(16 Sep '19, 16:45) Frederik Ramm ♦

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

*italic* or _italic_
**bold** or __bold__
link:[text](http://url.com/ "title")
image?![alt text](/path/img.jpg "title")
numbered list: 1. Foo 2. Bar
to add a line break simply add two spaces to where you would like the new line to be.
basic HTML tags are also supported

learn more about Markdown

Question tags:

import ×196
planet ×56

question asked: 16 Sep '19, 14:14

question was seen: 2,546 times

last updated: 16 Sep '19, 16:45

Full Planet Import Shortest Time

Follow this question

Related questions