NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hello,

After installing a fresh version of the overpass api on my private server, I have imported a planet file (http://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/planet/planet-latest.osm.bz2) and most of the features are running smoothly. However, running is_in area queries only returns some areas like countries, rather than more specific areas like buildings. Has anyone encountered a problem like this before? There doesn't seem to be any sort of documentation on this matter. I have not altered the areas.osm3s file and am running version osm3s_v0.7.50.

Example problem:

If I run this request from the public server:

http://overpass-api.de/api/interpreter?data=%0A%5Bout%3Ajson%5D%3B%0Ais_in(40.70523754289769%2C-74.05129551887512)%3Bout%3B%0A

(should return Liberty State Park as an area),

my custom server does not return Liberty State Park, but does return everything else. Does anyone know what the problem here could be?

Thanks to everyone.

asked 31 Aug '14, 23:46

gmeister4's gravatar image

gmeister4
608812
accept rate: 0%


I could import a 26GB pbf planet file into my local Overpass API instance on my laptop (4GB RAM, SSD, 16GB swap space) without much issue. Database import took around 38 hours, area creation around 6 hours. Total disk usage now is about 182GB for the database. Note: No attic data at this time (I don't need it right now).

For the area creation I used the original script, but had to increase the element-limit to "2073741824" - otherwise the script would terminate with a runtime error "Query run out of memory in "recurse" at line 255 using about 1157 MB" as you mentioned. This is not an OS level error but enforced by Overpass API itself.

I could not reproduce the missing tags as you describe it. However, I removed all "area*" files in the database directory after a failed area creation before trying again. Not sure, if this is really necessary. Just wanted to give it a clean start.

permanent link

answered 13 Sep '14, 18:13

mmd's gravatar image

mmd
5.7k15388
accept rate: 37%

edited 13 Sep '14, 18:13

2

This did it! You are a life saver mmd! I had tried increasing the element-limit to 3073741824 before, but this did not work. I cleaned up the old area files and used your recommended limit which did the trick.

For the record, the missing tags were arising from replacing the area rules file with the areas_low_ram.osm3s from the overpass misc folder, which would complete the batch runs but like I said, would leave missing tags from the area elements. Thanks again! You are like the patron saint of the overpass api!

(14 Sep '14, 19:51) gmeister4

Please check the following things:

  • does "way(27459219);out meta;" find the Liberty Park outline and has it a name tag and a tag "leisure=park", and is it version 25?
  • does the areas.osm3s in $DB_DIR/rules contain the rule for "leisure" (I know it's unchanged but just in case)
  • Are there two dispatcher processes running (one with --osm-base and one with --areas)?
  • What is the content of $DB_DIR/rules_loop.out (last 10 lines should be sufficient)?
permanent link

answered 01 Sep '14, 07:37

Roland%20Olbricht's gravatar image

Roland Olbricht
6.7k36489
accept rate: 36%

edited 01 Sep '14, 07:38

Thanks for the response,

  • "way(27459219);out;" finds Liberty Park, it has a name and leisure tag (but I do not have metadata so cannot see version number)
  • rules contains the leisure node
  • Both dispatcher processes are running
  • Last 9 lines of rules_loop.log are:

2014-08-31 22:49:36: update started 2014-08-31 23:54:41: update finished 2014-08-31 23:54:44: update started 2014-09-01 01:00:35: update finished 2014-09-01 01:00:38: update started 2014-09-01 02:05:22: update finished 2014-09-01 02:05:25: update started 2014-09-01 03:09:48: update finished 2014-09-01 03:09:51: update started

(01 Sep '14, 08:32) gmeister4

Any suggestions as to what could be going wrong? All other node/ way queries work that I have tried so far

(01 Sep '14, 15:19) gmeister4

At least it is nothing obvious. The area generation is suspiciously fast (should be closer to 7 hours than to 1.5 hours). So I would ask you to check

  • Does "area[leisure=park];out ids;" have any results?

  • When you look at $EXEC_DIR/bin/nohup.out, do you have a line containing the term "part 0, on line 257" (or line 255)? This would tell us whether the area generation terminates for some reason in between.

(01 Sep '14, 16:55) Roland Olbricht

Area[leisure = park] does return around 4500 id's, however, $EXEC_DIR/bin/nohup.out does not exist but you were right! in /root/nohup.out I have found After 1h4m32s: in "recurse", part 0, on line 255. Stack: 1 of 3884956 0 of 1 and also runtime error: Query run out of memory in "recurse" at line 255 using about 1157 MB of RAM.. My server has 4gb. Note: My server is running 99.9% RAM load, with 'top' telling me that osm3s_query is running heavy:

6898 root 20 0 725m 215m 872 R 94.2 5.3 19:51.32 osm3s_query

What can I do to rectify this? Thanks Roland.

(01 Sep '14, 18:34) gmeister4

Please replace the areas.osm3s in $DB_DIR/rules by the file http://overpass-api.de/misc/areas_low_ram.osm3s

It contains the same queries, but it generates areas immediately after each block. It is probably slower, but it should use less RAM.

A second approach would be to raise the number 1073741824 in the first line of areas.osm3s or its replacement. That is the soft upper limit of RAM the query should use. But it likely won't help much if the script is killed by the operating system instead of the scheduler, because it hits the hard limit of 4GB RAM minus other processes.

(01 Sep '14, 19:16) Roland Olbricht

I have managed to complete the first block in 8h 30m with the areas_low_ram. All of the areas are now showing up, but many of the areas do not have all their correct tags, such as the 'name' tag. Is this due to the modified script? Or should these tags appear after successive blocks?

(03 Sep '14, 09:56) gmeister4

Still not the correct tags

(04 Sep '14, 11:27) gmeister4
showing 5 of 7 show 2 more comments

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×483
×290
×213
×147
×69

question asked: 31 Aug '14, 23:46

question was seen: 8,452 times

last updated: 14 Sep '14, 19:51

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum