NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hello, what I'm trying to achieve is that I'd like to split a large OSM file into smaller OSM files which are going to be based on web mercator tiles at a particular zoom level, let's say 16.

I'm aware of osmium-tool CLI and I know I could use this command:

osmium extract --bbox 28.97369384765625,41.025499378313754,28.9764404296875,41.027571415339786 --strategy complete_ways -o smaller_tile.osm -f xml large_file.osm.pbf --overwrite

The issue is that I needed to run a python script that executes the command above with different bbox boundaries (using mercantile package) — and every individual osmium extract ... run is parsing the source large_file.osm.pbf again and again, which makes the whole process rather slow and taking more time than necessary.

So, I'd like to code a C++ tool myself on top of libosmium directly and produce multiple osm files for tiles while handing the source OSM file.

I think I can implement a osmium::handler::Handler and use osmium::apply(reader, handler) API. However, I don't really know how I need to handle node, way and relation entires coming through. Would it be enough to check entry's coordinate against a tile boundary and put it into a map keyed with tile xy and later somehow build and write OSM file while iterating over the main tile-entry map?

A super small and simple example would be really useful for me to understand how I can implement a simpler and less sophisticated version of osmium extract ... by myself.

I'm hoping to find a direction and potentially also shed light on this for other for future reference, thank you.

asked 02 Sep '23, 02:35

mfatihmar's gravatar image

mfatihmar
11112
accept rate: 0%

Why do you want to do this? There may be an existing tool for your end goal.

(02 Sep '23, 10:26) InsertUser

I'm trying to break a large OSM file of a city into smaller chunks in web mercator tiles at a particular zoom level, so that I then convert them into 3D OBJ files and stream over the internet (over a real-time network connection). It's a pretty custom workflow and instead of hosting a full-blown tile-server, all I need is static OSM and OBJ files built/baked let's say every month automatically. I have looked at OSM2World, osmosis, osmium etc. and I think I need to write some custom tools on top of lower-level libraries like libosmium. Thank you for asking though :)

(02 Sep '23, 13:37) mfatihmar

There is no "super small and simple example" because it is not a super small and simple problem you are trying to solve. You can look at the Osmium source code and find all the details in there.

I suggest a different approach: osmium extract can read a list of extracts to generate from a config file instead of getting the bounding box from the ocmmand line. You can auto-generate that config file. Be aware that there is a limit how many extracts you can generate in one go, though. That's just a question of memory usage. Every extract needs memory, see the man page for details. So for a practical use case you probably want to split the input file into, lets say 100 pieces and then run osmium extract again for each of the pieces to split them further.

permanent link

answered 02 Sep '23, 08:05

Jochen%20Topf's gravatar image

Jochen Topf
5.2k55074
accept rate: 31%

I had a look at osmium extract and considered --config something.json option but the limits are simply too low for me. On the C++ side, reading and processing a 50-100MB .osm.pbf file is not a big deal but osmium-tool set the memory and file limits simply too low for the pipeline I'm trying to build.

I also had a look at these two places related: - https://github.com/osmcode/osmium-tool/issues/265 - https://github.com/osmcode/osmium-tool/commit/353e9d565342af1ea703c56a15f41dc40732168a

Perhaps it's a bit complex than I initially anticipated but at least currently, I'm thinking of processing relations first, ways second and nodes the last to determine dependencies between them and create&use osmium::io::Writers to write those osmium objects/elements into files. (I hope this would work!)

It is a little bit hard for me to read osmium-tool codebase (command-extract) and follow its logic. I will, however, use osmium extract outputs to compare it against my implementation's output to validate my work.

I wanted to see if there are hints/tips/tricks for this but apparently no, I need to implement a proper tool on top of libosmium.

(02 Sep '23, 13:58) mfatihmar

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×710
×287
×38

question asked: 02 Sep '23, 02:35

question was seen: 1,733 times

last updated: 02 Sep '23, 13:58

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum