exporting boundaries from osm.pbf into csv file

Question

Hi!

I am trying to get boundaries from usa.osm.pbf file using "pyrosm" with the following command:

import pyrosm
# Initialize the OSM object 
osm = pyrosm.OSM("usa.osm.pbf") 
boundaries = osm.get_boundaries('all') 
boundaries.to_csv('usa.csv')

I have a PC with 256 GB ram, using Linux, and usa.psm.pbf file allocates 8.3 GB on disc. However, I encounter with memory issues. Ironically, 256 GB RAM is not enough to extract boundaries from 8.3 GB file.

How can I achieve my task in a different way using my luxury and expensive, but not sufficient resources?

Answer 1

2

You have still not explained in what way the resources are insufficient. I do not know about pyrosm but I suspect it will inefficiently try to load all nodes into memory, even those that are not needed for the particular task. It may be possible to first use the osmium command line utility with the tags-filter option to remove everything from the usa.osm.pbf that is not needed for a boundary extract, and then try pyrosm again.

Another option that will definitely work is importing the data into a PostGIS database with osm2pgsql and then using PostGIS SQL queries to extract the boundaries.

permanent link

answered 16 Jun '22, 14:20

Frederik Ramm ♦
82.5k●92●720●1273
accept rate: 23%

Thank you Frederik.

Could you explicitly show me here how I can do it, step by step? I have newly met with OpenStreetMaps.

Best,

(16 Jun '22, 14:25) hasan90

I would be happy if you could show PostGIS method. Thanks!

(16 Jun '22, 14:30) hasan90

Since administrative boundaries are almost always relations, the osmium approach would probably be (all on one line!)

osmium tags-filter 
  -o usboundaries.osm.pbf
  usa.osm.pbf
  r/boundary=administrative

Then you could try processing the (much smaller) resulting usboundaries.osm.pbf with your existing workflow. Alternatively, install osm2pgsql and follow the instructions on https://osm2pgsql.org/doc/manual.html to load your file into the database, then you can execute a SQL query like

SELECT name,admin_level,way
FROM planet_osm_polygon
WHERE boundary='administrative';

to find boundaries.

(16 Jun '22, 14:32) Frederik Ramm ♦

Frederik, thank you for your answer! I highly appreciate it.

I would like to get 'all' boundaries, not only administrative.

(16 Jun '22, 14:39) hasan90

exporting boundaries from osm.pbf into csv file

Follow this question

Related questions