Hello, After installing a fresh version of the overpass api on my private server, I have imported a planet file (http://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/planet/planet-latest.osm.bz2) and most of the features are running smoothly. However, running Example problem: If I run this request from the public server: (should return Liberty State Park as an area), my custom server does not return Liberty State Park, but does return everything else. Does anyone know what the problem here could be? Thanks to everyone. asked 31 Aug '14, 23:46  gmeister4 |
2 Answers:
I could import a 26GB pbf planet file into my local Overpass API instance on my laptop (4GB RAM, SSD, 16GB swap space) without much issue. Database import took around 38 hours, area creation around 6 hours. Total disk usage now is about 182GB for the database. Note: No attic data at this time (I don't need it right now). For the area creation I used the original script, but had to increase the element-limit to "2073741824" - otherwise the script would terminate with a runtime error "Query run out of memory in "recurse" at line 255 using about 1157 MB" as you mentioned. This is not an OS level error but enforced by Overpass API itself. I could not reproduce the missing tags as you describe it. However, I removed all "area*" files in the database directory after a failed area creation before trying again. Not sure, if this is really necessary. Just wanted to give it a clean start. answered 13 Sep '14, 18:13  mmd edited 13 Sep '14, 18:13 |
Please check the following things:
answered 01 Sep '14, 07:37  Roland Olbricht edited 01 Sep '14, 07:38 Thanks for the response,
2014-08-31 22:49:36: update started 2014-08-31 23:54:41: update finished 2014-08-31 23:54:44: update started 2014-09-01 01:00:35: update finished 2014-09-01 01:00:38: update started 2014-09-01 02:05:22: update finished 2014-09-01 02:05:25: update started 2014-09-01 03:09:48: update finished 2014-09-01 03:09:51: update started (01 Sep '14, 08:32) gmeister4 Any suggestions as to what could be going wrong? All other node/ way queries work that I have tried so far (01 Sep '14, 15:19) gmeister4 At least it is nothing obvious. The area generation is suspiciously fast (should be closer to 7 hours than to 1.5 hours). So I would ask you to check
(01 Sep '14, 16:55) Roland Olbricht Area[leisure = park] does return around 4500 id's, however, $EXEC_DIR/bin/nohup.out does not exist but you were right! in /root/nohup.out I have found 6898 root 20 0 725m 215m 872 R 94.2 5.3 19:51.32 osm3s_query What can I do to rectify this? Thanks Roland. (01 Sep '14, 18:34) gmeister4 Please replace the areas.osm3s in $DB_DIR/rules by the file http://overpass-api.de/misc/areas_low_ram.osm3s It contains the same queries, but it generates areas immediately after each block. It is probably slower, but it should use less RAM. A second approach would be to raise the number 1073741824 in the first line of areas.osm3s or its replacement. That is the soft upper limit of RAM the query should use. But it likely won't help much if the script is killed by the operating system instead of the scheduler, because it hits the hard limit of 4GB RAM minus other processes. (01 Sep '14, 19:16) Roland Olbricht I have managed to complete the first block in 8h 30m with the areas_low_ram. All of the areas are now showing up, but many of the areas do not have all their correct tags, such as the 'name' tag. Is this due to the modified script? Or should these tags appear after successive blocks? (03 Sep '14, 09:56) gmeister4 Still not the correct tags (04 Sep '14, 11:27) gmeister4 showing 5 of 7 show 2 more comments |
This did it! You are a life saver mmd! I had tried increasing the
element-limitto 3073741824 before, but this did not work. I cleaned up the old area files and used your recommended limit which did the trick.For the record, the missing tags were arising from replacing the area rules file with the
areas_low_ram.osm3sfrom the overpass misc folder, which would complete the batch runs but like I said, would leave missing tags from the area elements. Thanks again! You are like the patron saint of the overpass api!