Skip to content

download-ignore-retry-after-header#44

Open
amith-ananthram wants to merge 1 commit intoallenai:mainfrom
amith-ananthram:download-ignore-retry-after-header
Open

download-ignore-retry-after-header#44
amith-ananthram wants to merge 1 commit intoallenai:mainfrom
amith-ananthram:download-ignore-retry-after-header

Conversation

@amith-ananthram
Copy link
Copy Markdown

@amith-ananthram amith-ananthram commented Jul 31, 2025

  • pass through kwargs that set timeout in the function call
  • ignore retry after header as some URLs set them arbitrarily far in the future, causing the download script to hang eg:
(base) amith@Amiths-MacBook-Pro-4 metrics % curl -I https://www.avialogs.com/media/k2/items/cache/b53ad912c7950b115492257a5e8e3636_M.jpg
HTTP/1.1 503 Service Unavailable
Date: Thu, 31 Jul 2025 23:02:44 GMT
Server: Apache/2.4.58 (Ubuntu)
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Referrer-Policy: strict-origin-when-cross-origin
**Retry-After: Sat, 10 Aug 2025 00:00:00 GMT**
Last-Modified: Tue, 24 Jun 2025 15:42:32 GMT
ETag: "3f2b-638532d34c015"
Accept-Ranges: bytes
Content-Length: 16171
Connection: close
Content-Type: text/html

@amith-ananthram
Copy link
Copy Markdown
Author

@chrisc36 not sure how to request a review but wanted to share this fix; happy to iterate on it if you'd like more sophisticated logic too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant