IDS does not implement the RFC 9110 protocol correctly

[andygotz]

At ESRF we would like to use the aria2 download tool for its ability to download large volumes reliably. The IDS server is preventing this because it does not implement the RFC 9110 protocol correctly. Two scenarios have already been identified when aria2 stops because of an error given to the IDS. The problem is aria2 exchanges with IDS messages about Content-Range which IDS misinterprets and throws an error which then stops the tranmission. This can happen during a long transmission or at the end of a short/long transmission.

Here is a typical example of running aria2 (note the ids url below is temporary and not valid anymore):

aria2c.exe --file-allocation=none --log=aria2.log https://ids.esrf.fr/ids/getData?sessionId=d6e94b53-89ff-4d20-b460-42e790ba9985"&"datafileIds=2222697526"&"outname=78.36um_S-22-16_brain_complete-organ_pag-0.14_0.34_jp2_.zip

Here is feedback from Paul Millar who helped us identify the problem:

(1) The IDS service only accepts 'Range' request that match the RE:

bytes=(\\d+)-

Here's the code:

ids.server/src/main/java/org/icatproject/ids/IdsService.java

Line 370 in 9188381

rangeRe = Pattern.compile("bytes=(\\d+)-");

Aria2 sends the HTTP request header:

Range: bytes=10405591040-10971021401

The header value doesn't match the string, as Java does a whole-string match.

The partial download works with IDS if I modify the request to include only the offset in the byte-range; i.e.,

Range: bytes=10405591040-

Technically, aria2 code could be updated to send the offset-only requests. However, this is really just a work-around for IDS' limited support for the "Range" HTTP header.

Incidentally, I opened a similar issue against Aria2 when we were working under EOSC-Future:

https://github.com/aria2/aria2/issues/2051

Marjolaine patched the IDS server at ESRF to accept Range messages above specifying two numbers but then it failed on Content-Range:

(2) The problem occurred when interrupting the transfer (with ~50% of the file download) and then asked aria2 to download the remainder.

The problem appears to be the lack of "Content-Range" HTTP response header.

Per RFC 9110 https://www.rfc-editor.org/rfc/rfc9110#status.206

15.3.7.1. Single Part

If a single part is being transferred, the server generating
the 206 response MUST generate a Content-Range header
field, describing what range of the selected representation
is enclosed, and a content consisting of the range.

The 206 response from IDS is missing the "Content-Range" header.

Without this header in the response, aria2 appears to be assuming a zero-byte offset with the range taken from the "Content-Length" response header. This explains the aria2 error message:

Invalid range header. Request: 827785216-1638331884/1638331885, Response: 0-810546668/810546669

You will probably have more success if IDS were to be updated so any 206 response includes the required "Content-Range" header.

HTH,
Paul.