Hello there,

I am very inexperienced in the whole topic, however I managed to set up a tile server using postgresql and renderd following this tutorial: https://switch2osm.org/manually-building-a-tile-server-18-04-lts/

Since I want to use the map for cosmetic purposes only (application only displays the surrounding map; basic shape of streets, buildings and parks), I wonder if the osm.pbf files only contain these raw shape data or if there is a lot of meta data connected with it, to allow querying the postgres for special places etc. If this is the case, is there a way to get only the "shape files" which would lead to the following:

  1. faster osm2psql imports (this is a real bottleneck if one wants to scale a tile server horizontally)
  2. less disc space necessary
  3. faster access times

An other question I was really interested in is, if there are any scripts out there which do the steps described in the tutorial above automatically. I tried to make one by myself, but switching users (to postrges and back) does not really seem to work in a script.

I would be really happy if anyone could clear things up for me, maybe I completely misunderstood the kind of files that exist and how to use them.

Greetings,

Erik

asked 24 Jan, 14:00

Erik1988osm's gravatar image

Erik1988osm
21225
accept rate: 0%

edited 24 Jan, 14:40


One answer to the "cosmetic purposes only" part of the question might be to significantly cut down the data that load load into the database (which obviously will mean that it is smaller and will load quicker). This diary entry and the one that it lnks to describe doing that for boundaries; you'd want to include a few more features I expect but the principal should still work.

It still might not be the best solution for what you want though; something involving tailored shapefiles might be better, and I'm sure there's a write-up of that somewhere.

permanent link

answered 24 Jan, 14:46

SomeoneElse's gravatar image

SomeoneElse ♦
32.1k63332751
accept rate: 15%

Thank you. Am I right with my understanding, that osm.pbf files contain much more than just the graphical structure of the map? Do the "shapefiles" (https://www.geofabrik.de/en/data/shapefiles.html) only contain the graphics? And if yes, can I somehow import just these in a postgresql to use with renderd afterwards? I guess for renderd to allow the query there must be at least the positioning data correlated with them to work properly. Do you know any hoster, which offer these minimalistic map graphic/position osm.pbf data? As far as I can overlook your link, the changes are only made on the style, the backlaying data remains still unchanged (big and slow).

(24 Jan, 14:57) Erik1988osm

Am I right with my understanding, that osm.pbf files contain much more than just the graphical structure of the map?

Yes - the contain all the data in OSM, including many things that any particular map is unlikely to display. However to be clear is is no one "the map" - there are lots of different representations of OSM data. This is a fun map showing lots of different map styles.

"Shapefile" is just a file format, just as .osm.pbf is, and the page you link to explains a bit about what Geofabrik have put in their shapefiles.

As far as I can overlook your link, the changes are only made on the style, the backlaying data remains still unchanged (big and slow).

No, as the linked diary entry says "for boundaries I created a version of that that used osmium, osmconvert and osmfilter to select only boundaries from the data and remove non-boundary tags that might render too".

(24 Jan, 15:05) SomeoneElse ♦

Right. So what my solution would be is to not modify the style, as I did to hide certain thinks like name tags, banner etc., but to use osmfilter on a planet.osm.pbf and only keep certain informations of interest. This would probably lead into a way smaller file (?). But I wonder if I could still apply a style to it, since some fields would be missing. Does the osm2pgsql just skip those?

(24 Jan, 15:20) Erik1988osm

But I wonder if I could still apply a style to it, since some fields would be missing. Does the osm2pgsql just skip those?

It depends - you do have the option to cut down the columns created in the database at data load (using a "style" file), and if you do that then yes, a rendering style that expects those columns to be present will give errors. You do however have the option to create all the columns and just not load data into them - that shouldn't cause the renderer to fail.

Given that you want a map that looks different to the OSM Carto one, you probably want to experiment with modifying the style as well.

(24 Jan, 15:40) SomeoneElse ♦

Thank you very much for your help, it made certain things clear to me. I created a summary below on what steps are necessary, as far as I understood for now.

(24 Jan, 15:50) Erik1988osm

Can you explain what you mean with tailored shapefiles a little more in detail?

(24 Jan, 16:02) Erik1988osm

Can you explain what you mean with tailored shapefiles a little more in detail?

One of the more GIS people here is probably better placed to do that, but essentially a shapefile is just a file format. The page you linked to before offers special purpose shapefiles for e.g. power infrastructure, for example. There may be someone out there creating "minimal" shapefiles for just major roads, towns, that sort of thing.

(24 Jan, 17:04) SomeoneElse ♦
showing 5 of 7 show 2 more comments

Re "doing it automatically" this diary entry describes how to do it all using Docker.

permanent link

answered 24 Jan, 14:22

SomeoneElse's gravatar image

SomeoneElse ♦
32.1k63332751
accept rate: 15%

As a summary to what my understanding was, the following steps will lead to a custom file, reduced in its size (and with this probably raise the performance as well):

  1. Get a planet.osm.pbf
  2. Examine the file, to get knowledge about the field names you require:

    ./osmfilter bremen.o5m --out-count

      28199 highway
      22794 building
      21365 name
      18754 created_by
      17751 source
      11842 addr:housenumber
      11835 addr:street
       5925 addr:postcode
       5724 foot
       5543 addr:city
    
  3. Apply the filter

    ./osmfilter norway.osm --keep="highway=primary =secondary waterway=river" >streets.osm

  4. Convert the output file with osmconvert if necessary

  5. Update your style xml (?)

  6. Import the custom file with osm2pgsql

permanent link

answered 24 Jan, 15:49

Erik1988osm's gravatar image

Erik1988osm
21225
accept rate: 0%

edited 24 Jan, 15:51

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×206
×158
×42
×27

question asked: 24 Jan, 14:00

question was seen: 566 times

last updated: 24 Jan, 17:22

powered by OSQA