Data download stuck

[caichaoxiang]

Hi, when I download some data, such as PixMoPoints and PixMoAskModelAnything, it always gets stuck when the download is almost complete. I changed different networks and repeated the download command, but the same error still occurs.

[amith-ananthram]

@caichaoxiang I ran into this issue too -- upon inspection of a culprit URL, I found that the retry logic needs to be amended to ignore "retry after" designations from URLs as they can be arbitrarily far in the future. for example:

curl -I https://www.avialogs.com/media/k2/items/cache/b53ad912c7950b115492257a5e8e3636_M.jpg
HTTP/1.1 503 Service Unavailable
Date: Thu, 31 Jul 2025 23:02:44 GMT
Server: Apache/2.4.58 (Ubuntu)
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Referrer-Policy: strict-origin-when-cross-origin
Retry-After: Sat, 10 Aug 2025 00:00:00 GMT
Last-Modified: Tue, 24 Jun 2025 15:42:32 GMT
ETag: "3f2b-638532d34c015"
Accept-Ranges: bytes
Content-Length: 16171
Connection: close
Content-Type: text/html

opened a PR with the fix here: #44