Support non-tar'd .gz files as a package format

[LikeLakers2]

Hi! Whilst trying to add some binstall metadata to taplo-cli, I came across an issue. Specifically, taplo-cli is released in a non-tar'd .gz format for all platforms (except Windows, which uses .zip).

Unfortunately, I found that cargo-binstall has no supported pkg-fmt option for just .gz. Looking over the formats that cargo-binstall supports, it seems to assume that if .gz is used, then we'll also be using a .tar.

With cargo-binstall already using some dependencies to extract .gz files, I would think that including support for non-tar'd .gz files would be trivial to add. You could probably even add the same sort of support to .bz2, .xz, and .zst files.

So if it isn't too unreasonable, perhaps this could be done?

[NobodyXu]

Yes I think it can be done relatively easily, though I don't have time to add that now.

If you are willing to open a PR, I am more than happy to review and merge it

[LikeLakers2]

@NobodyXu I'd be willing to make a PR.

However, I don't have much knowledge of cargo-binstall's internals. So it might take a few days while I acquaint myself with the codebase.

[NobodyXu]

No worries!

The PkgFmt is defined in binstalk-types and shared among all binstalk packages, and the guessing of PkgFmt from filename

binstalk-downloader handles the actual download and unpacking, once new PkgFmt is added, updating binstalk-downloader to handle the new variant.

You can use async-compression's gz decoders

[LikeLakers2]

@NobodyXu So I managed to implement it... but forgot that single-file archive formats (like .gz, .bz2, etc.) don't actually use filenames. So cargo-binstall never actually finds the correct binary inside .gz files, unless the .gz file is named the expected name.

Unfortunately, I'm unsure how to get the expected name - so I don't think non-tar'd .gz files would be possible without additional work.

Additionally, I found that if a repository has both a .tar.gz and a .gz file that match, the async requests race to see which one returns first - meaning that cargo-binstall would no longer always go after the same download. (which is important in this case, as .tar.gz can hold multiple files, where .gz can only hold one)

At this point, I'm unsure how to proceed. I feel it may be better to let you try to handle it (since I don't really want to touch anything more than what I already have).

My work thus far is at https://github.com/LikeLakers2/cargo-binstall/tree/project/add/non-tar-formats (or you can download the sole commit here) if you want to use any part of it. (In case it needs said for legal stuff: I give you permission to do whatever you wish with those changes, with or without credit.)

[NobodyXu]

but forgot that single-file archive formats (like .gz, .bz2, etc.) don't actually use filenames. So cargo-binstall never actually finds the correct binary inside .gz files, unless the .gz file is named the expected name.

If it's a single file, then it shouldn't be a problem right?

We just use gz for compression, not using it as an archive?

if a repository has both a .tar.gz and a .gz file that match

I think this shouldn't happen, for a crate to have two archive formats with different content, and not specify pkg-fmt manually.

We don't need to consider this case, at the very most we can add a warning message.

[LikeLakers2]

If it's a single file, then it shouldn't be a problem right?

It is - because formats like gz, xz, and bz2 don't store a filename alongside the file. To compensate for this, extraction software assumes that the single file inside is named the same as the containing archive (except for the .gz, .xz, etc. suffix).

This means that if the uploaded .gz file isn't named the same as the binary we're looking for, cargo-binstall won't find the binary it's looking for.

[NobodyXu]

To compensate for this, extraction software assumes that the single file inside is named the same as the containing archive (except for the .gz, .xz, etc. suffix).

Yeah that how I think it should work

This means that if the uploaded .gz file isn't named the same as the binary we're looking for, cargo-binstall won't find the binary it's looking for.

Yeah I think it sounds good?

I.e. for cargo-binstall it would be cargo-binstall.gz

[NobodyXu]

To compensate for this, extraction software assumes that the single file inside is named the same as the containing archive (except for the .gz, .xz, etc. suffix).

Yeah that how I think it should work

This means that if the uploaded .gz file isn't named the same as the binary we're looking for, cargo-binstall won't find the binary it's looking for.

Yeah I think it sounds good?

I.e. for cargo-binstall it would be cargo-binstall.gz

[LikeLakers2]

Yeah I think it sounds good?

I.e. for cargo-binstall it would be cargo-binstall.gz

Not really, because then the file name doesn't tell us what OS (Linux/Darwin/etc.) or architecture (aarch64/x86/powerpc/etc.) the contained binary is for.

If we try to rename it to include such information (i.e. cargo-binstall-x86_64.gz), then the file inside also gets renamed (i.e. to cargo-binstall-x86_64) - and thus cargo-binstall starts complaining about the binary not being found.

[NobodyXu]

Oh right I forgot that.

On the second thought the bin pkg-fmt also has the same problem, and that isn't a problem because for installation we use the binary name recorded in Cargo.toml in a crate, so it also won't be a problem for .gz files

[ilyagr]

I also encountered this when trying to make binstall work with https://github.com/drdo/redu/releases, which uses plain bz2 archives.

See also #2126

Glad to see people are already looking at it!