NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

0
1

I have this local setup of overpass in a 32core 16Gb-ram server. My requests are not processing parallel, when i trigger multiple threads of the same request, the total time taken is = (time taken to complete one request * no of threads). Can someone point me towards what am missing here!! So i would like to know if overpass handle's requests parallel or sequential ?

Tnx in advance - AG

asked 20 May '16, 10:13

Arun%20Gowtham's gravatar image

Arun Gowtham
0111114
accept rate: 0%

edited 20 May '16, 11:15


As you can see in the munin stats for the official server, the server handles about 400-700 requests/minute. That for sure wouldn't be possible by processing a single request at a time. Also, the CPU utilization clearaly shows that all 8 cores all completely used up.

Conclusion: processing is parallel by default, each request is handled by a different Apache CGI process up to a certain limit.

Note that there's a limit of one request per IP at the same time on the official instance. That most likely doesn't apply if you're running your own instance, unless you set it up explicitly.

Unless you're adding more detail on your setup and your query, it's impossible to tell what kind of issues your might see on your local instance. It might also be that you're hitting some I/O limit or experiencing CPU saturation, too many parallel requests for your machine, internal scalability issues in the dispatcher, wrong/insufficient configuration of your web server, your load driver may not work properly. There are literally dozens of different influencing factors.

permanent link

answered 20 May '16, 11:40

mmd's gravatar image

mmd
5.7k15388
accept rate: 37%

edited 20 May '16, 11:41

thanx for the info @mmd I'll debug my web server configuration's and look for issues anywhere else.

(20 May '16, 12:03) Arun Gowtham

hi @mmd

Here's the case am trying to solve:

My query: [out:json];area[name='Tamil Nadu'];node(area)[amenity='atm'];out count;

The time taken for this single request is :

1 new Thread started

Time taken for req :1 : 1936 (ms)

Whereas when i trigger 10 threads of the same request i get such result,

10 new Threads started

Time taken for req :1 : 1870 (ms)

Time taken for req :7 : 1869 (ms)

Time taken for req :2 : 1870 (ms)

Time taken for req :3 : 1870 (ms)

Time taken for req :8 : 1868 (ms)

Time taken for req :6 : 1869 (ms)

Time taken for req :5 : 1947 (ms)

Time taken for req :10 : 3588 (ms)

Time taken for req :4 : 3590 (ms)

Time taken for req :9 : 3950 (ms)

Is this normal, what i am expecting is all those 10 requests should have completed approximately around 1870 (ms), but it took additional time.

When i increase the thread count, say to 50, the total time taken goes upto 14 seconds.

When increased ever further to 500, i start to get 504 error's for nearly 370 of the requests and the time taken for entire set including the errors is 35 seconds

Please help me sort out this issue, is this functionality normal or have i configured wrong?! :(

Thanks-AG

(20 May '16, 12:39) Arun Gowtham

So it seems to scale fine up to 7 requests in parallel. Your system has 8 cores and surely has some other minor stuff to do while processing your requests. The remaining 3 requests take twice as long since they have to wait for the other requests to finish. Sounds fine to me. However I don't know any internals about Overpass. But just bumping up the allowed number of threads will usually have a negative impact on the system since they will also introduce additional overhead and will lead to processing more quests in parallel than the system can handle, slowing down each individual request.

(20 May '16, 14:21) scai ♦

@scai actually my machine is 32 core cpu...

(20 May '16, 14:25) Arun Gowtham

Oh, sorry. I've taken the 8 cores mentioned in mmd's answer for your system. Anyway, as already explained by mmd there can be multiple other reasons like limited I/O. You can for example check your wa (=wait) in top.

(20 May '16, 14:31) scai ♦

yeah,i get you, i just wanted to know if what i stated above is not normal in overpass, so that i would start to debug elsewhere..

(20 May '16, 14:35) Arun Gowtham
showing 5 of 6 show 1 more comments

We were facing the same problem, using an unofficial Docker image. It probably had a bug in its Nginx configuration. Updating this image solved it.

permanent link

answered 15 Jun '20, 15:06

flogoe's gravatar image

flogoe
76227
accept rate: 0%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×483
×85

question asked: 20 May '16, 10:13

question was seen: 3,940 times

last updated: 15 Jun '20, 15:06

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum