|
After I have downloaded 11 small countries in interval of 9 minutes and then 9 bigger countries in interval of ~ 90 minutes from Geofabrik download site, my requests for country/state download are rejected with reason "Too Many Requests" (HTTP 429). HTTP response does not contain HTTP header "Retry-after" and I cannot find on Geofabrik Web site information about allowed rate of download service requests. Please, let me know what allowed rate of download requests is so I can change my application to obey that rule. Thank you! |
|
The Geofabrik download server has a long history; it started as primitive download page where you could download a Germany extract, for the benefit of users in Germany who couldn't handle the whole planet. The scope has meanwhile increased, but the basic idea is still that this server is for those who only require a small part of the planet. Users who want data for all of Europe for example are encouraged to actually download the Europe file and use a readily available tool like The current server configuration allows a bulk download of 40 files (not counting diff updates, state.txt, HTML pages etc), after which the bucket refills at a speed of 1 per hour. This is not advertised though because it can change at any time, and I don't want people to write scripts that depend on this limit being unchanged. As a rule of thumb, if you don't download more than 24 files a day, you should always be fine. If your use case requires downloading more, you should think about downloading a larger file. Thank you Frederik so much for these valuable information. Since I need entire planet I will not base my solution on the assumption that I can use you Web site more intensively than you have explained in this post. BTW, I sent the first email on June 22 at 10:50 AM PST (10 days ago) and the last thing I want is to be a consumer with extremely short patience. Actually, I am very thankful for what you are doing for all of us.
(03 Jul '18, 07:54)
MiroslavK
Hello Frederik, I'm maintining https://tile.openstreetmap.be/. I launched the update today and discovered that the download of the files was really slow (first time I notice it). Our tileserver covers BeNeLux, North of France, and West of Germany ; so I download the subset I need, merge them and import the merged result in the database. So far I could download the 10 files I need in a few minutes but today even the first file (https://download.geofabrik.de/europe/belgium-latest.osm.pbf) is tresholded at around 30KB/s and pauses really often. Estimated time of download for belgium-latest.osm.pbf is around 60 minutes. Is there an issue ? Did I miss something ? Such a slow download really makes the update of our tileserver almost impossible. What would the best way to do what we need to do (download Belgium and surroundings) so it would be fast enough for us but also not too heavy on your server ? Thanks a lot !
(09 Jul '18, 09:53)
jbelien
2
There are three different download servers and none of them is throttled in any way but it is not impossible to have one of them being a little slow once in a while. It's certainly not done on purpose.
(09 Jul '18, 10:52)
Frederik Ramm ♦
No worries, download is complete now. Thanks for your answer :)
(09 Jul '18, 11:30)
jbelien
Hi guys, I've been having the same issue - too many requests - when downloading maps . It occurred at 4 maps being downloaded, 10 more to go. I wonder which sleep delays I better set between downloads in order to get 14 maps and not fail in the middle.
(24 Oct '18, 11:05)
TattiQ
I am looking at alternative mirrors, https://ftp.gwdg.de/pub/misc/openstreetmap/download.geofabrik.de/ seems to have only Europe as a whole and Germany.
(24 Oct '18, 11:28)
TattiQ
1
If you get a "429" message after downloading just 4 files, then someone else must be using the same IP as you and making requests too (e.g. if you're accessing from a university through a shared proxy or so). If you want you can email your IP address to info@geofabrik.de and we'll have a look in the log files. Downloading 14 files in one day should never be a problem.
(24 Oct '18, 12:15)
Frederik Ramm ♦
showing 5 of 7
show 2 more comments
|
did you try to contact them directly ? There is some contact information on their website. I know there are some Geofabrik employees that read here a well, but in general, it is better to contact commercial providers directly if you have questions about their service.
I did send email to that address two times so far but it seems to me the emails do not reach technical people and that is why I came here hopping Geofabrik technical people visit this forum.
I saw that Geofabrik provides commercial services but the download service is for free. Regardless of being free or payed service, the providers should publish the limits and, based on the posts in this forum, it seems to me that is common practice of other providers of free services (returning HTTP 429 without clear explanation of how many requests represent "Too Many Requests").
I am aware of the cost of providing such services and that is why I am very thankful for free services they provide to all of us and if I can help in any way to improve usage of those services I would do that happily.
The only email I can see in my Geofabrik inbox from you is from 20:13 yesterday evening. Either earlier ones have been lost, or your patience is extremely short even for an OpenStreetMapper. Also there is no way you can have run into a 429 error after making just 20 downloads.