Libosmium - How to filter and store tags from a POI?

Question

Hi

I'm new to OSM and Libosmium. My goal is to filter certain POI e.g. all the libraries worldwide with the location and certain tags and store it as a json to later processing (e.g. search nearest library). As source file sI use OSM extracts from http://download.geofabrik.de/

To be able to process it locally I started to look into Libosmium even though I'm a newbie with cpp. I'm using Ubuntu 20.04 and was able to compile Libosmium (which was already great). Under /examples/ I found a pretty close example osmium_amenity_list.cpp https://github.com/osmcode/libosmium/blob/master/examples/osmium_amenity_list.cpp which I want to adop to my needs. Could someone please help me with the following:

1) How can I filter on amenity=library? My current solution:

const char *amenity_type = node.tags()["amenity"];
if (amenity_type)
{
    if (strcmp(amenity_type, "library") == 0)
    {...

Is there a way to query amentiy=libray within the first line?

const char *amenity_type = node.tags()["amenity"];

2) Is there a way to loop through all available fields e.g. for nodes I want to extract as well not only "name" + "location (lat, lon)", but also e.g. id, addr:country, addr:city, addr:street, addr:housenumber, ...

3) Instead of printf() of print_amenity() I want to store it as json file. I'm still new to cpp (I'm more familiar with python, where I would e.g. store each entry as a dict() within a list() and later convert the list into a json or dataframe). e.g. I would like to query later either with cpp or python the json for the closest libraries within a certain range via haversine.

-> How would I adopt it to instead of print store it in a json or similar format? What I found was e.g. the proposal to use a map in combination with a tuple and a vector.

#include <map>
#include <tuple>
#include <vector>
using KeyType = std::tuple<int, int>;
using ValueType = std::vector<KeyType>;
std::map<KeyType, ValueType> Nodes;

Not sure if this is the best way to be used with Libosmium (or put different how should I replace print_amenity() to be able to store and later further process it)?

Thank you very much

Answer 1

Rather than learning C++ for this project and especially because you are already familiar with Python I suggest the following approach: Use the osmium command line tool to do a rough filtering of the input, for instance with osmium tags-filter input.osm.pbf n/amenity=library -o output.osm.pbf. This will give you an OSM file with only those nodes that have a tag amenity=library which is much much smaller than what you had before and because Osmium is written in C++ it is quite fast. After that use PyOsmium for the rest, more detailed filtering, conversion to JSON or whatever else you want to do. It will be a bit slower than processing in C++, but development will be much easier.

To answer your questions (somewhat): 1. The approach you are using is correct, 2. You can loop over all tags with for (auto const &tag : node.tags()) {...}. 3. You'll need a JSON library, it depends on the library how best to drive that. The answer to that goes well beyond something I can answer quickly here.

Answer 2

@Jochen: Thank you very much for your answer. I like you proposed approach and I followed on it.

a) Thanks to the amazing "osmium command line tool" I was able to quickly filter the pbf file for amenity=library (I only extended the filter to also include ways and relations with "nwr/" instead of "n/" only for nodes.

b) For the further processing I already tried to use PyOsium before and was struggling to get it working to extract tags and store them (https://github.com/osmcode/pyosmium/issues/208). I was trying to follow the recommendations in https://github.com/osmcode/pyosmium/blob/master/doc/intro.rst but I keep getting "RuntimeError: Node callback keeps reference to OSM object. This is not allowed." That's the reason why I started to try to do it with Libosmium, because I was stuck with this error in python. When I try to run it with the following python program (which is a copy of the example from 2 like mentioned in the github issue 1, adopted for "amenity=library" and with the file http://download.geofabrik.de/europe/germany-latest.osm.pbf from Geofabrik.

import osmium
from pathlib import Path

class HotelHandler(osmium.SimpleHandler):
    def __init__(self):
        super(HotelHandler, self).__init__()
        self.hotels = []

    def node(self, o):
        if o.tags.get('amenity') == 'library' and 'name' in o.tags:
            print(str(o.tags['name']))
            self.hotels.append(o.tags['name'])

def main():
    data_file = Path('/home/..../germany-latest.osm.pbf')
    h = HotelHandler()
    h.apply_file(data_file)

    print(sorted(h.hotels))

main()

I get the first 2 entries

Stadtbibliothek Weinheim

Deutsche Nationalbibliothek

appended to the list (and also print out) and afterwards the program stops with "Node callback keeps reference to OSM object. This is not allowed."

My technical set-up: Ubuntu 20.04, Python 3.8.10, IDE PyCharm 2022.2

May I ask you, if you could help me with what is going wrong in the python program? Thank you very much

Libosmium - How to filter and store tags from a POI?

Follow this question

Related questions