NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hello,

I would like to setup a small private OSM Server in the cloud for testing an application. To achieve this, I've launched an AWS EC2 t2.micro instance and followed the installation steps from the Official OSM Install Guide.

In general the installation worked fine, at least for very small countries. I've successfully installed Andorra (Geofabrik-Link) and Faroe Islands (Geofabrik-Link) and was able to run queries on the instance.

When trying to do same with a little bit larger country, in this case Austria (Geofabrik-Link), i receive a "broken pipe" error:

Reading XML file ...terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc src/bin/init_osm3s.sh: line 44: 29086 Broken pipe bunzip2 < $PLANET_FILE 29087 Aborted (core dumped) | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION

Can someone help me to get this fixed?

The resources of my EC2 instance currently are the following: - Ubuntu Server 16.04 LTS (HVM) - 1 GB RAM - ~25-30GB of free disk space (using AWS EBS)

My simple guess is that the resources are too low, especially the Memory of 1GB. Before I upgrade to more resources (I want to avoid costs as much as possible) and test again, I would like to how likely this would fix my problem? Also, can anybody tell me how much RAM and Disk Space I approximately need for a ~900MB OSM-file like Austria?

Update 1: In the meantime I've upgraded my machine and now have the following resources:

  • Ubuntu 16.04 LTS
  • 2vCPUs
  • 7.5 GB RAM
  • ~50 GB of free SSD disk space

I tried to run the Austria Import again. Unfortunately it failed again (shortly after starting the import) with a slightly different error:

Reading XML file ... elapsed node 1559520430. ./exec/bin/init_osm3s.sh: line 44: 18530 Broken pipe $ 18531 Killed | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION

Again, I was able to import the very small country export Faroe-Island. I also tried importing a "medium" large country in terms of file size: Belgium. Running the import with Belgium showed no errors. (I manually canceled the import after around 5-10mins).

Is 7.5GB RAM still not enough? Or can it be another problem?

Update 2: Now I also tested importing a larger country, Czech Republic with a file size of ~1.1GB. Interestingly, the import again seemed to run without problems and I saw the import progress without any errors (i.e. many lines with "Reading XM L file...elapsed node xxxx. Flushing to databse ... done.) I cancelled after around 10min.

To summarize my import approaches:

  • country | filesize | status
  • austria | ~900MB | failed (both on 1GB RAM and 7.5GB RAM)
  • faroe | ~3MB | worked (both)
  • belgium | ~500MB | worked (on 7.5GB RAM)
  • czechrep | ~1.1GB | worked (on 7.5GB RAM)

I can only guess, but since importing all countries (even larger one's) except for Austria seemed to work, I suspect there might be something wrong with the export hosted at Geofabrik? Anyone had similar problems before? Someone knows another server that hosts country files, or is there a way how I can find out if the file is "OK"?

asked 14 Nov '17, 13:13

hiasel's gravatar image

hiasel
21113
accept rate: 0%

edited 15 Nov '17, 10:21

1 GB RAM sounds pretty low. Check dmesg for out of memory (OOM) messages.

(14 Nov '17, 13:18) scai ♦

Thanks for quick response. Could you help me how exactly i shall use the dmesg command? I've just executed it and simply searched for "OOM", but can't find anything. Or shall I run this command (in parallel) while I do the database import?

(14 Nov '17, 13:24) hiasel

Run it afterwards: dmesg | grep -i "out of memory". If there is no output then too few memory is not your issue.

(14 Nov '17, 13:43) scai ♦

hm. just executed dmesg | grep -i "out of memory" right after calling init_osm3s.sh and there is indeed no output. The error remains. The full error message btw. starts with "Reading XML file ...terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc" (i've updated that in the question).

(14 Nov '17, 14:31) hiasel
1

std::bad_alloc definitely sounds like having too few memory. The program doesn't get killed by the kernel but terminates itself since it can't allocate enough memory. Get more RAM or try a smaller map extract.

(14 Nov '17, 15:32) scai ♦

allright, then I will upgrade my machine and post the result as soon as I got it running. Thanks a lot already for your help :)

(14 Nov '17, 15:36) hiasel

I just tried running the import again with an upgrade machine (See my updated question). Still no success. I wonder if 7.5GB is still not enough? According to the Install Guide "1 GB of RAM and sufficient swap space for a small extract or a development system." To me, austria still seems like a "small" extract compared to others, so I thought with ~8GB i'm on the safe side. Is it possible that there is something wrong with the file-extract of austria at geofabrik? (I've downloaded a fresh copy each time i tried running the import, so it shouldnt be my fault)

(15 Nov '17, 09:39) hiasel
1

You could try one of the other extracts.

(15 Nov '17, 10:29) scai ♦
1

Thanks for pointing me to the list of extracts. In the end I used a custom extract from: http://extract.bbbike.org/. For my project I only need a part of Austria, i.e. my custom extract file size now is only ~150MB. Importing this custom extract worked without problems and I can perform queries on my cloud instance :)

(16 Nov '17, 10:04) hiasel
showing 5 of 9 show 4 more comments

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×483
×61
×9
×4
×1

question asked: 14 Nov '17, 13:13

question was seen: 2,406 times

last updated: 16 Nov '17, 10:04

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum