NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hi, I want to set up the car profile data, but it is now taking 3 days and still at 100% Graphs (so way more to go) and I am thinking, maybe my swap and stxxl is just set up inappropriately.

I have the following system

  • 2 HDs 4 TB SATA 6 Gb/s 7200 rpm HDD Enterprise-Class
  • 64 GB DDR4 RAM
  • Intel® Core™ i7-6700 Quad-Core Skylake incl. Hyper-Threading

HD1 has the

  • system
  • programs
  • osm.pbf file on it
  • the extracted data must go on this one
  • partition with 32G swap

HD2 mostly is empty (but the space is going to be needed for some other big data)

My extract approach was to set up another swap of 300G on HD2 and stxxl of 300G on HD2. But it seems to be too slow. iotop has some reading and some writing activity on it, but nothing major. htop does not show any significant cpu usage. I start with 8 processes.

  1. Is there a better way to set up the additional swap and stxxl (maybe the swap on HD1 and only the stxxl on HD2)?
  2. If I could move the files and target locations of the extracted data as wanted, would it be even better to maybe do .pbf on HD1, all swap and stxxl on HD1 and extracted data on hd2(and then just copy it to HD1)?

Thanks

asked 02 Sep '17, 09:10

wordli's gravatar image

wordli
11225
accept rate: 0%

How large is the input file (.osm.pbf file)? Do you try to process the whole planet?

You might have a look at https://lists.openstreetmap.org/pipermail/osrm-talk/2017-July/001460.html for RAM requirements.

(04 Sep '17, 09:41) Nakaner

yes, I am processing the whole planet (37G)

(04 Sep '17, 19:00) wordli

Processing the full planet with the default car profile will take more than 150 GB of RAM. In your case, with 64 GB of RAM and 32 GB of swap, I'd expect it to die at some point; you'd need 96 GB of swap to even make it work and then it would take very long.

One way of saving memory is splitting the planet up into disjunct areas, of course at the cost of not being able to route between them later on. Or you could use a different routing software that has a lesser memory footprint (but might then be slower to run).

64 GB of RAM should be enough to run the routing once computed, so another option would be to rent a large-memory Amazon machine for a day and have the extract processed there.

permanent link

answered 04 Sep '17, 21:29

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
82.5k927201273
accept rate: 23%

That is unfortunate that you cannot extract on a machine that could actually run the program. But good idea with the outsourced processing. Thanks!

(07 Sep '17, 08:13) wordli

Note that older versions of OSRM have a smaller memory footprint: depending on the features you require, you may find that 4.9.1 fits your needs.

(07 Sep '17, 15:41) Richard ♦

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×4
×1
×1

question asked: 02 Sep '17, 09:10

question was seen: 2,033 times

last updated: 07 Sep '17, 15:41

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum