NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hi!

I am trying to get boundaries from usa.osm.pbf file using "pyrosm" with the following command:

import pyrosm
# Initialize the OSM object 
osm = pyrosm.OSM("usa.osm.pbf") 
boundaries = osm.get_boundaries('all') 
boundaries.to_csv('usa.csv')

I have a PC with 256 GB ram, using Linux, and usa.psm.pbf file allocates 8.3 GB on disc. However, I encounter with memory issues. Ironically, 256 GB RAM is not enough to extract boundaries from 8.3 GB file.

How can I achieve my task in a different way using my luxury and expensive, but not sufficient resources?

asked 16 Jun '22, 13:18

hasan90's gravatar image

hasan90
11113
accept rate: 0%

edited 16 Jun '22, 14:27

And what is your question?

(16 Jun '22, 14:02) Frederik Ramm ♦

Sorry Frederik, I have just added the question.

(16 Jun '22, 14:08) hasan90

You have still not explained in what way the resources are insufficient. I do not know about pyrosm but I suspect it will inefficiently try to load all nodes into memory, even those that are not needed for the particular task. It may be possible to first use the osmium command line utility with the tags-filter option to remove everything from the usa.osm.pbf that is not needed for a boundary extract, and then try pyrosm again.

Another option that will definitely work is importing the data into a PostGIS database with osm2pgsql and then using PostGIS SQL queries to extract the boundaries.

permanent link

answered 16 Jun '22, 14:20

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
82.5k927201273
accept rate: 23%

Thank you Frederik.

Could you explicitly show me here how I can do it, step by step? I have newly met with OpenStreetMaps.

Best,

(16 Jun '22, 14:25) hasan90

I would be happy if you could show PostGIS method. Thanks!

(16 Jun '22, 14:30) hasan90

Since administrative boundaries are almost always relations, the osmium approach would probably be (all on one line!)

osmium tags-filter 
  -o usboundaries.osm.pbf
  usa.osm.pbf
  r/boundary=administrative

Then you could try processing the (much smaller) resulting usboundaries.osm.pbf with your existing workflow. Alternatively, install osm2pgsql and follow the instructions on https://osm2pgsql.org/doc/manual.html to load your file into the database, then you can execute a SQL query like

SELECT name,admin_level,way
FROM planet_osm_polygon
WHERE boundary='administrative';

to find boundaries.

(16 Jun '22, 14:32) Frederik Ramm ♦

Frederik, thank you for your answer! I highly appreciate it.

I would like to get 'all' boundaries, not only administrative.

(16 Jun '22, 14:39) hasan90
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×129
×42
×12
×3

question asked: 16 Jun '22, 13:18

question was seen: 1,363 times

last updated: 16 Jun '22, 14:39

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum