NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hi,

I just got a dedicated server for our planet osm2pgsql database. It has 32 GB RAM, two Opteron processors (6212) (16 cores in total), a software RAID1 for the system and a software RAID0 for the database. The database disks are 10k Raptor drives, so nothing too slow. I tested an import of the planet during the weekend and I think the performance of the import could be better, especially when reviewing Frederiks SotM presentation in 2010 where the import is done within some hours (regard the slim planet import whith R0, -C8000).

The planet.osm.bz2 is on the RAID1, the import command was:

osm2pgsql -d osm -s -S /home/brfr/default.style -C 16000 -U postgres -H localhost -k --number-processes 16 planet-latest.osm.bz2

so actually plenty of cache available. During the process I observed that the node processing speed never got over 80k/s which is somewhat low I think for that machine. Using top I watched the osm2pgsql process and it was always at 100%. In order to find out if there's a disk speed limit I used iostat and the values I saw let me assume that the drives were never near to their limits. So I tend to believe that I run into a CPU limit while processing the XML. Unfortunately the import process was interrupted. There were too many connections on the database (the limit was set too low by the pgtune tool). So I don't have any final numbers, just the XML processing stats below. So finally my question. Can someone confirm that Opterons suck at XML processing? Does someone have experience with Opterons and a osmpgsql import? What's your node processing speed?

The switch2osm site has a quote which shows a nearly tripled speed (200k/s), that's mainly why I ask here for experience values from other planet "importers" ...

My stats for the processing:

Node stats: total(1403664658), max(1684121047) in 18315s
Way stats: total(129971557), max(156125599) in 14955s
Relation stats: total(1342628), max(2092829) in 18429s

Many thanks

Frank

asked 02 Apr '12, 09:04

frabron's gravatar image

frabron
3611411
accept rate: 16%

edited 02 Apr '12, 09:06

Thanks all, I am downloading the planet.pbf and will post the results probably on wednesday and also in the wiki. Right now I want to test if my Intel (i4) at home performs better than the Opteron at work on processing nodes

(02 Apr '12, 20:03) frabron

Ok, I downloaded great-britain.osm.bz2 and the node processing speed definitely faster than the Opterons speed. Right now it processes nodes with 93,6k/s and the ways between 10 and 11k/s. My testing setup is simple, I use a Virtualbox with the latest Ubuntu server release and osm2pgsql from source. The host is a i4 2400 Intel processor and 8GB RAM with 4GB given to the machine. Osm2pgsl runs with -C 2048 ...

(02 Apr '12, 20:59) frabron

Today I started a new import with the latest planet.osm.pbf. Node import speed is at approx. 140k/s, already much better than the XML import. One thing I notice though, now the postgres process is at 100%. It is the

      query_start          | procpid | waiting |           substring

-------------------------------+---------+---------+-------------------------------

2012-04-04 10:44:12.982371+02 | 63618 | f | COPY planet_osm_nodes FROM ST

action that now is limiting a faster import. Disks are nowhere at their limits and osm2pgsql is between 30% - 60% CPU usage

(04 Apr '12, 10:48) frabron

FYI: I added my server and results (~50hrs) to the benchmark table ...

(10 Apr '12, 08:14) frabron

Dear frabron, I'm so glad you have solved you problem. And now I run into the same qusetion like yours, so your advice is using the .pbf format data to import instead of the .bz2 ? Do you have some method to accelerate the importing speed of the bz2 file? Does the command " --number-processes 16" in you command line count a lot?

With many thanks!! David

(03 Dec '14, 02:55) studiousdavid

First of all, forget my performance figures, they are two years old and that's a very long time in OSM!

You can use the .pbf planet instead of the .osm.bz2, this will save the time used for bz2 decompression and XML parsing. However, I would expect over 80% of processing time to be spent building geometries and indexes, so even if your system should be slow while reading the file, that should not impact overall performance too much.

permanent link

answered 02 Apr '12, 09:27

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
82.5k927201273
accept rate: 23%

You might check this wiki page collecting several osm2pgsql performance measurements (and add yours):

http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks

permanent link

answered 02 Apr '12, 12:21

Pieren's gravatar image

Pieren
9.8k2083157
accept rate: 15%

edited 02 Apr '12, 22:49

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×196
×56
×28

question asked: 02 Apr '12, 09:04

question was seen: 12,455 times

last updated: 03 Dec '14, 18:44

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum