Replace websave with curl-based downloads to prevent corruption#746
Merged
Replace websave with curl-based downloads to prevent corruption#746
Conversation
Gateway-level HTTP compression can corrupt binary/zip downloads when MATLAB's websave auto-decompresses content-encoded responses, causing "not a tar file" / invalid archive errors on Linux (Mac is unaffected). Route the download through ndi.cloud.api.files.getFile with useCurl=true, matching the pattern already used by didsqlite.do_openbinarydoc.
Gateway HTTP compression corrupts binary downloads when websave auto-decompresses the response. Route through getFile with useCurl=true.
Gateway HTTP compression corrupts binary downloads when websave auto-decompresses the response. Route through getFile with useCurl=true.
The API gateway serves binary artifacts (.zip for bulk document downloads, .nbf.tgz for epoch binaries) with Content-Encoding: gzip. websave was transparently decoding these on Mac, which masked the issue there; plain `curl -L -o` does not decode and writes the compressed bytes to disk, producing "invalid ZIP file" / "not a tar file" errors. Adding --compressed makes curl advertise Accept-Encoding and decompress the response, matching websave's prior behavior.
Our cloud payloads are already compressed archives (.zip, .nbf.tgz). Layering HTTP-level gzip on top has produced corrupt archives on both Mac (stream unzip fails) and Linux (untar fails) even with --compressed, likely because streaming decoders misbehave on already-compressed content. Explicitly request identity encoding so the gateway delivers the file bytes as-is. Add -f so curl exits non-zero on HTTP errors instead of writing an error body to the destination file (which would then masquerade as a corrupt archive).
stevewds
approved these changes
Apr 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces
websavewith curl-based downloads across three download modules to prevent HTTP content-encoding corruption. Thewebsavefunction automatically decompresses HTTP responses, which can corrupt binary files when the gateway applies compression at the HTTP level.Key Changes
websavewithndi.cloud.api.files.getFile(..., 'useCurl', true)and improved error handling with retry logic that captures the last error messagewebsavewith curl-basedgetFilecall with proper error handlingwebsavewith curl-basedgetFilecall with proper error handlingImplementation Details
ndi.cloud.api.files.getFile()function with the'useCurl'parameter set totruesuccess_dreturn value and converts error responses to strings for consistent error messaginglastErr) to report in the final timeout error, fixing a potential bug whereME.messagecould be undefinedhttps://claude.ai/code/session_01HMnM1qnDBgGdjSqSnjTfsV