I need to have a local tile server set up because you guys block MOBAC, and I need offline maps for geocaching. (These are for my consumption, and I don't need uninterrupted service. But it's also nice to be able to point Leaflet at them, and maybe tweak them. And I do need tiles for offline use, for both Cachebox and Cachewolf.)

Now, I’ve imported Belgium, to try this out, and the entire disc use is about 8 GiB after installing everything needed, where /var/lib/postgresql is about 5½ GiB. I do not have unlimited space here, only about (currently) 16 GiB more. (I could possibly try to get this set up on another box though I'd need space estimates there too. It could probably do 400 GB.)

I have two ways I can go forward from here:

① drop DB, recreate DB, import Europe extract from Geofabrik

② combine NL/BE/LU/DE/AT/CH extracts where I need them

The problems here are:

osm2pgsql does not want to do ②; I’m being told to combine the extracts with osmosis before importing. However, how do I keep the combination up-to-date later?

How much disc space would I need for ① ? I assume I can then throw the files from http://download.geofabrik.de/europe-updates/000/000/ to osm2pgsql to keep it updated?

Could I possibly import just the Europe changesets in the ② scenario, too?

osm2pgsql works. I didn’t use osmosis yet, and I don’t quite see what parts of its functionality I need for all this (GIS newbie here).

asked 24 Jan '15, 21:09

mirabilos's gravatar image

mirabilos
11113
accept rate: 0%

edited 25 Jan '15, 12:55

Hm, this sounds like I need 200-300 GiB total for Europe uncut, and to stay away from flat nodes due to lack of upgradability. Still, the question stays open, in case someone else has better figures, and (more importantly) instructions/recommendations.

(24 Jan '15, 22:02) mirabilos

Flat nodes don't impact upgradability but their benefit is greatest for full planet imports.

Since you do not need a continuously running tile server - you can afford downtimes while updating - the most space-efficient way to do what you want is

  1. if your area of interest is rectangular, goto 2; else determine the area you want and create a "poly" file for it (e.g. draw polygon in josm, save as .osm, use osm2poly.pl - or use another method)
  2. download europe-latest.osm.pbf from Geofabrik
  3. use osmosis (or osmcut or osm-history-splitter) to cut out the area of interest from the Europe file
  4. import the resulting file with osm2pgsql using the --slim and --drop options which leads to a non-updatable database and makes the database unusable during import
  5. whenever you want to update, repeat steps 1-4.

Alternatively, if you do want uninterrupted service, follow steps 1-3 above and then

  1. import the resulting file with osm2pgsql using --slim
  2. when you want to update, repeat steps 1-3 above and then
  3. use osmosis with the --derive-change option to compute the difference between the new file of interest and the last one and save to an .osc file
  4. use osm2pgsql with --append to load that .osc file

There are other options, like consuming the Europe diffs from the Geofabrik site, but that would lead to unnecessary data in your database that you would have to remove from time to time.

permanent link

answered 25 Jan '15, 00:37

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
68.3k806231067
accept rate: 24%

edited 27 Jan '15, 09:04

2

If you want to trim diffs to your poly of bounding box before applying, have a look at trim_osc.py in https://github.com/Zverik/regional . That will greatly reduce database growth due to "out of area" updates.

(25 Jan '15, 09:28) SomeoneElse ♦

Thanks, this mostly sounds like a plan. (Indeed, I don't need uninterrupted service.) But it seems I have no way to estimate the size? Also, “use osmosis to…” is still a bit hard to look through, I don't grok its documentation well. Mh. I'll wait a bit to see what other answers there are.

(25 Jan '15, 12:50) mirabilos
2

osmosis --read-pbf in.osm.pbf --bbox left=a right=b top=c bottom=d --write-pbf out.pbf is the command line for cutting out a rectangle, and osmosis --read-pbf new.osm.pbf --read-pbf old.osm.pbf --derive-change --write-xml-change diff.osc.gz is the command line for producing a diff.

(27 Jan '15, 09:11) Frederik Ramm ♦
1

Thanks for this recipe. This was one of those recipes I was looking for actually :-)

(27 Jan '15, 14:19) nordie69

In answer to "How much disc space would I need for ① ?", I suspect that 300Gb would be a bit low, currently. I don't have a Europe extract maintained currently, but when I last did (a couple of years ago) my recollection is that it needed 250-300Gb, so the requirement would be larger now.

However, I'm sure that someone must have more up to date figures.

permanent link

answered 25 Jan '15, 09:35

SomeoneElse's gravatar image

SomeoneElse ♦
31.7k63325737
accept rate: 15%

Hm okay. Thanks for the estimate. (Maybe I'll just get another box to play mapserver.)

(25 Jan '15, 12:50) mirabilos
1

A world-wide planet needs about 450 GB in PostGIS, and the planet.osm.pbf is 27 GB. The europe.osm.pbf is about half that size so 250 GB will probably be sufficient. This is for an updatable --slim database; --drop should save quite a bit of that. All just a wild guess though since of course the relation between PBF size and database size is not necessarily constant.

(27 Jan '15, 09:09) Frederik Ramm ♦

At the start of this you say "I need to have a local tile server set up because you guys block MOBAC, and I need offline maps for geocaching.".

I'm guessing that the issue here is lack of data coverage and cost of mobile data when you're out and about.

However, maybe it's worth taking a step back and asking if lots of individual map tiles is the best way to hold and display data? I don't know what you're displaying data on, but if it's a phone there are quite a few options (Osmand's offline layer may work for you if you're using an Android phone, and plenty of other options are available).

permanent link

answered 25 Jan '15, 09:44

SomeoneElse's gravatar image

SomeoneElse ♦
31.7k63325737
accept rate: 15%

The issue here is the price (and speed) of GPRS coverage in foreign countries, yes. I am using Cachebox for Windows Mobile which serves my Geocaching needs and also seems to be the best moving map application for Windows Mobile.

(25 Jan '15, 12:53) mirabilos
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×208
×193
×69
×45

question asked: 24 Jan '15, 21:09

question was seen: 4,756 times

last updated: 27 Jan '15, 14:19

powered by OSQA