504 error when requesting data using wget

Question

I am using wget to request data on the national borders of a few countries in Western Europe from the Overpass API. I use the following request:

wget "http://overpass-api.de/api/interpreter?data=[out:xml][timeout:3600];(relation(45.0,-6.0,56.0,15.5)["boundary"="administrative"]["admin_level"="2"];node(r)->.admincentre;way(r);node(w););out;" -O res.osm

However, each time I get the following error response after a couple of minutes (the first part is in Dutch ;) ):

HTTP-verzoek is verzonden; wachten op antwoord... 504 Gateway Timeout

Is there any way to resolve this error? I set the timeout to 3600 seconds, but it seems to be ignored. Is that on purpose and is my request too large? If this is the case, are there other options to request my data when sticking to the Overpass API?

Thanks in advance.

Answer 1

0

wget has its own timeout. Try to specify the same timeout via --read-timeout=3600. The default read timeout is 900 seconds according to my manpage.

permanent link

answered 04 Feb '16, 08:07

scai ♦
33.3k●21●309●459
accept rate: 23%

1

hmm, I think the 504 is sent from the server because an upstream server (possibly the overpass interpreter) has not answered yet. See http error codes. So, wget's own timeout should not have mattered - yet.

(04 Feb '16, 09:00) aseerel4c26 ♦

You are probably right. If a read timeout occurs then wget prints a different error message.

(04 Feb '16, 11:40) scai ♦

I added the timeout flag of wget to my request. However it doesn't seem to solve my problem. Thanks anyway. :)

(04 Feb '16, 18:49) Steijn

Answer 2

The Overpass website gives the following information:

504 Gateway Timeout is sent if the server has already so much load that the request cannot be executed. In most cases, it is best to try again later. Please note that the server decides this based on the timeout and maxsize parameters of the request, so smaller values for them may also make the request acceptable.

So to me the problem seems to be related to the complexity of my request and the amount of requests made to the server at that moment. Cutting the requests in parts, as RM87 suggested earlier, indeed solves the problem.

However, is it possible to show some kind of traffic diagram of the Overpass instances? That way I can do my requests, I have several of them and also more complex, in the more quiet hours. :)

504 error when requesting data using wget

Follow this question

Related questions