Right now we are using Bing Map Web Services API for Map/geocoding/reverse geocoing at our application. Our application is installed on premise and so we are evaluating some on premise replacement. Based on current research, OpenStreetMap should be a good candidate for this purpose, however I found out this from OpenStreetMap dowloading data

The entire planet is a huge amount of data. Common tools like Osmosis or various import tools for database imports and converters usually need several days to process it, even on fast computers.

This is almost 27 GB compressed or 370 GB uncompressed XML.

The file size is quite huge for an on premise installation and plus long time to load data into database.

We don't need all layers of the data, just wonder if there is any simple approach we would strip the data to just show the road Map.

asked 21 Oct '13, 18:44

windfly's gravatar image

windfly
16112
accept rate: 0%


The exact space required will vary with what are you import and how you import it. I wouldn't regard the decompressed XML size as too relevant because a) virtually no one decompress the XML for the planet to disk and b) you should use the .pbf formatted data, not the .osm.bz2 formatted data because PBF is a much faster format to process, and smaller too.

I've also updated the wiki page with more accurate speeds.

If you want to render tiles on-demand with the standard setup you need an osm2pgsql database. I posted two months ago details of the space required. It is possible to run a rendering server off of one 7200 RPM hard drive, but it won't be super-fast.

Rendering and geocoding are normally run on separate servers because they need independent databases - a database structured for one is not structured for the other. I don't have the same experience with nominatim as I do with osm2pgsql and rendering, but poldi, the server powering nominatim.osm.org, has a 1TB database.

A rendering database takes under a day to import on reasonably powerful hardware.

permanent link

answered 22 Oct '13, 08:45

pnorman's gravatar image

pnorman
2.4k52140
accept rate: 19%

Have a look at osmfilter ... maybe needs osmconvert first do work via a speedy data format

permanent link

answered 21 Oct '13, 19:52

stephan75's gravatar image

stephan75
12.5k454209
accept rate: 6%

The easiest way to slim down OSM data is by region instead of by feature class, so if you can work with e.g. "just Europe" or "just North America" then you'll already save a lot.

Pre-filtering by feature is possible (using e.g. the Osmosis or osmfilter tools) but it is not straightforward. Some in-depth knowledge will be required to actually define what exactly you want - these tools don't take "just the road map" for an answer. Does that include the traffic light, the barrier, the turning circle?

A typical "tile server" is based on a PostgreSQL database. You could set up a full rendering server on a big machine connected to the Internet, and then export from that a - regional or thematic - subset to deploy on your on-site server. Since the on-site subset wouldn't have to be updatable, this would incur significant space savings.

Another option is to pre-produce ready-made map tiles on a big machine and store them on the on-site machine. This works if you only need a regional extract or only a limited amount of detail, e.g. the world down to zoom level 12 or so.

permanent link

answered 21 Oct '13, 20:13

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
74.5k866731152
accept rate: 24%

1

Between the unwanted history, type of data, and type of use, you can reduce the needed size drastically. The initial whole-world import takes a lot of time, but keeping it up to date is easyer.

Do your initial importing and filtering on a smalish country, like Denmark. You'll be able to iterate faster and get a proper estimate of what you will need once you add France, then Europe, then the rest of the world.

(21 Oct '13, 22:03) Vincent de P... ♦

thanks, at this point, it won't be on one country. it is the whole world since this would be installed on customer on premise env and we don't know what country they would need

(22 Oct '13, 22:20) windfly

Hi I solved this and wrote an Medium article how to do this. Hope it will help:

https://medium.com/dac-technology-blog/openstreetmap-map-size-reduction-guide-43372bbf38c9

permanent link

answered 17 Aug, 15:24

JacekDAC's gravatar image

JacekDAC
11112
accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×22

question asked: 21 Oct '13, 18:44

question was seen: 11,118 times

last updated: 17 Aug, 15:24

powered by OSQA