Hello, I would like to setup a small private OSM Server in the cloud for testing an application. To achieve this, I've launched an AWS EC2 t2.micro instance and followed the installation steps from the Official OSM Install Guide. In general the installation worked fine, at least for very small countries. I've successfully installed Andorra (Geofabrik-Link) and Faroe Islands (Geofabrik-Link) and was able to run queries on the instance. When trying to do same with a little bit larger country, in this case Austria (Geofabrik-Link), i receive a "broken pipe" error:
Can someone help me to get this fixed? The resources of my EC2 instance currently are the following: - Ubuntu Server 16.04 LTS (HVM) - 1 GB RAM - ~25-30GB of free disk space (using AWS EBS) My simple guess is that the resources are too low, especially the Memory of 1GB. Before I upgrade to more resources (I want to avoid costs as much as possible) and test again, I would like to how likely this would fix my problem? Also, can anybody tell me how much RAM and Disk Space I approximately need for a ~900MB OSM-file like Austria? Update 1: In the meantime I've upgraded my machine and now have the following resources:
I tried to run the Austria Import again. Unfortunately it failed again (shortly after starting the import) with a slightly different error:
Again, I was able to import the very small country export Faroe-Island. I also tried importing a "medium" large country in terms of file size: Belgium. Running the import with Belgium showed no errors. (I manually canceled the import after around 5-10mins). Is 7.5GB RAM still not enough? Or can it be another problem? Update 2: Now I also tested importing a larger country, Czech Republic with a file size of ~1.1GB. Interestingly, the import again seemed to run without problems and I saw the import progress without any errors (i.e. many lines with "Reading XM L file...elapsed node xxxx. Flushing to databse ... done.) I cancelled after around 10min. To summarize my import approaches:
I can only guess, but since importing all countries (even larger one's) except for Austria seemed to work, I suspect there might be something wrong with the export hosted at Geofabrik? Anyone had similar problems before? Someone knows another server that hosts country files, or is there a way how I can find out if the file is "OK"? asked 14 Nov '17, 13:13 hiasel
showing 5 of 9
show 4 more comments
|
1 GB RAM sounds pretty low. Check
dmesg
for out of memory (OOM) messages.Thanks for quick response. Could you help me how exactly i shall use the dmesg command? I've just executed it and simply searched for "OOM", but can't find anything. Or shall I run this command (in parallel) while I do the database import?
Run it afterwards:
dmesg | grep -i "out of memory"
. If there is no output then too few memory is not your issue.hm. just executed dmesg | grep -i "out of memory" right after calling init_osm3s.sh and there is indeed no output. The error remains. The full error message btw. starts with "Reading XML file ...terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc" (i've updated that in the question).
std::bad_alloc
definitely sounds like having too few memory. The program doesn't get killed by the kernel but terminates itself since it can't allocate enough memory. Get more RAM or try a smaller map extract.allright, then I will upgrade my machine and post the result as soon as I got it running. Thanks a lot already for your help :)
I just tried running the import again with an upgrade machine (See my updated question). Still no success. I wonder if 7.5GB is still not enough? According to the Install Guide "1 GB of RAM and sufficient swap space for a small extract or a development system." To me, austria still seems like a "small" extract compared to others, so I thought with ~8GB i'm on the safe side. Is it possible that there is something wrong with the file-extract of austria at geofabrik? (I've downloaded a fresh copy each time i tried running the import, so it shouldnt be my fault)
You could try one of the other extracts.
Thanks for pointing me to the list of extracts. In the end I used a custom extract from: http://extract.bbbike.org/. For my project I only need a part of Austria, i.e. my custom extract file size now is only ~150MB. Importing this custom extract worked without problems and I can perform queries on my cloud instance :)