End-to-end integrity of crates in registries

[withoutboats]

I have been working on a proposal which would add a feature to cargo to make registries trust free - that is, to verify the integrity and authenticity of crates downloaded from a registry like crates.io even if crates.io were compromised.

I don't have an RFC yet, but I have a sketch & I thought I'd post it here. In brief, we automatically sign crates on publication and verify the signature on download.

First, we need a concept of a source of truth about user identities to cargo. For crates.io, the current source of truth is GitHub (crates.io does not manage its own authentication). Such a source of truth needs to have a feature which allows users to publish a public key that can be used to verify their signature. GitHub today allows users to publish GPG keys to verify their commits (though we don't need to actually use GPG, just format our public keys according to the OpenPGP spec - I wrote a crate to do this already for ed25519 keys).

We adjust the cargo login flow to generate a new key pair and publish the public key to GitHub - this happens entirely client side, crates.io never gains permission to publish GPG keys. The private key is saved in the user's .cargo directory, and used to sign crates published from this machine. Key revokation is done by deleting the key from GitHub.

In the registry, we additionally track the signature and the means to identify the public key (in this case, the info needed to request it from GitHub). When downloading crates, we verify the signature in the registry. We display to the user (published by GitHub user @withoutboats) or something similar, providing them the guarantee that this user did actually publish this data, and the data has not been modified since.

Critically, the registry plays no role in distributing public keys. Though I've so far discussed this in terms of distributing them through GitHub, the system can be flexible to allow additional sources of identity in the future, which would display different information.

Why not TUF?

1. TUF was not designed with an external authority about identity in mind, and so it involves the registry managing the ultimate source of truth for identities (the master key). We already delegate responsibility for managing user identity to a third party (GitHub).
2. TUF is opt-in, and has a high cost for opting in. Most users would probably not opt in.

In other words, adopting TUF would not provide much security for most users. All security would still depend on our security practices (in managing the master key), and only a small handful of crates would even see the gains.

The high opt-in cost has its own advantages: only users who are serious about key management would be likely to do it. However, users already have to trust the security practices of the authors of crates they depend on, and there's not really a way to avoid that.

Future extensions

Beyond the basic proposal I've sketched out, there are many ways the security of this system could be improved further:

1. We could adopt threshold signatures from TUF for crates with multiple owners, providing a higher degree of security (this would require a significant UX revamp).
2. We could adopt a trust-on-first-use system for ownership, to help catch e.g. withoutboots publishing a crate that actually belongs to withoutboats (there are some serious challenges I haven't worked out here about backwards compatibility and our github teams feature).
3. We could add options to require signatures and refuse to build crates without them, influencing resolution.

cc @rust-lang/cargo.

[withoutboats]

Also re TUF integration: in theory, an implementation of TUF could become an alternative "source of identity" eventually, as an alternative to using your GitHub account.

[dperny]

One of the problems that TUF solves is the issue of a malicious attacker distributing valid but out of date packages. Even an out of date package has a valid signature, and an attacker can present an old, insecure package to you as if its the latest version. If you're not cross checking and have no external idea of what the latest package version is, you could end up using insecure software.

I guess cross-checking against Github solves the problem? But then you need a more complex integration with Github's API, to determine what the latest stable version is. WDYT?

[withoutboats]

@dperny cargo uses the registry index to track what the most recent version of a package is, though there's no cryptographic verification of the index right now.

[withoutboats]

though there's no cryptographic verification of the index right now.

Its occurred to me that we should also start signing commits to the index. Here's a sketch:

• We add to the index format's config.json a list of committer keys.
• Every time we pull, we verify that every commit was signed by one of the keys.

[est31]

One of the problems that TUF solves is the issue of a malicious attacker distributing valid but out of date packages. Even an out of date package has a valid signature, and an attacker can present an old, insecure package to you as if its the latest version.

Its occurred to me that we should also start signing commits to the index

An attacker who has gotten access to the github repo where the index is stored can just force push to an older version of the index. This can be fixed however by looking at the date of the commit and requiring that it is at at most X old.

In general, getting some way of index signing onto the way is a very good idea, especially as it allows for third party mirrors outside of github. Your suggestion doesn't give much additional security for that though as it puts the list of keys into the config.json. It is probably easier to handle than baking the keys into cargo and then wanting to change it in the future when old cargo versions are still around :). DNSSEC has done that and got that precise problem :).

To the main point of the actual signature of the .crate files. What about having the users sign the json files of the registry? This would have several advantages:

• Suppose user x has gotten a new key and wants to re-sign all older versions of their crate. If the crate has some kind of an integrated signature, the user would have to download every single version of their crate, sign it, then re-upload it. That is against the immutable-by-design nature of crates.io. It would also break the sha256 hashes in all Cargo.lock files, and to viewers of the index it would look like some malicious edit.
• Transparency. Every change would be public and accountably visible on the public git index. If you mirror crates.io you would immediately be able to do the change yourself.
• It would verify the fact that some version of a crate has been yanked/not yanked with the key.

Also at this point it might be a smart moment to suggest a change in the way check sum creation works for .crate files: one issue I have observed with the way we are doing it right now is that you can't improve compression of the .crate files without changing the check sum, even though you don't touch any of the content. Also it is hard to have storage backends that do deduplication in order to reduce size. Therefore I suggest that check sums are not on the bare .crate files but instead on files inside those crate files that contain a hash of every file in the file. This suggestion is a bit orthogonal to what you probably target here @withoutboats but it would be cool if this could somehow be coordinated. I can also open a separate RFC if you want.

[withoutboats]

Your suggestion doesn't give much additional security for that though as it puts the list of keys into the config.json.

But a commit that would edit the config.json would only be accepted by end user cargo if it were signed by a key already in the config.json before the edit.

To the main point of the actual signature of the .crate files. What about having the users sign the json files of the registry? This would have several advantages:

I'm a little uncertain what you mean here. My intent is that the user signs the .crate file but the signature is stored in the index - that is, we're talking about using a detached signature.

To be more concrete, in my sketch, the objects in the index would gain these additional fields:

{
    "authority": "github-pgp",
    "publishkey": "withoutboats/1CC70310BE3912D5",
    "signature": "/* a hex encoded ed25519 signature */"
}

But the data signed to generate the signature would be the contents of the .crate file.

[est31]

that is, we're talking about using a detached signature.

I see. Everything is fine then!

[matthieu-m]

Could you please clarify how revocation is supposed to work?

At the moment, I have the impression that a famous user (such as @BurntSushi) revoking or rotating their keys would cause the same issue as NodeJS' infamous left-pad pull: if it becomes impossible to verify their crates, then it's exactly the same as their crate not existing any longer.

Or worse, users get used to ignoring "untrusted" warnings because revocations/rotations are so frequent that crates months old are always untrusted anyway.

[withoutboats]

@matthieu-m the implication of what you're saying is quite right - once a key is revoked, we can't hard fail on those crates or the ecosystem will be in ruins.

However, I'll note there are also a huge number of crates which are unsigned already, and those have to keep building. We cannot have "require valid signatures" as the default setting.

If the public key is revoked, the crate just does not display that it was published by any particular users. End users will have to make a choice about what to do in that situation. Someday we will probably support a flag to fail in that situation.

However, revocations should not be particularly frequent - only when a user believes their private key has been compromised. You can rotate keys without revoking the old one by deleting your local private key without deleting the public key. Users may be identified by any number of public keys published to their GitHub accounts, not one particular key.

(However, if there is a valid public key and a valid signature and the signature isn't valid, we should hard fail because that is clear evidence of tampering.)

[raggi]

TUF was not designed with an external authority about identity in mind, and so it involves the registry managing the ultimate source of truth for identities (the master key). We already delegate responsibility for managing user identity to a third party (GitHub).

You could still do this with a TUF root. When a user publishes a new version of a crate, if the users signing key changed, you would check against the users github key, and if the key is new (but the user auth is successful), then you would revoke the old key, and publish with the new kew. Working revocation model, and still delegated auth.

TUF is opt-in, and has a high cost for opting in. Most users would probably not opt in.

Again, this is a design choice for your implementation, and it needn't be. What if the process for setting up TUF when you publish was to push a TUF key to your github account somewhere. Done. The receive side, after this is setup, should not be opt-in at all. You can add a special marker in the targets that is explicit about the entire set of unsigned targets, and disallow any future unsigned targets.

In other words, adopting TUF would not provide much security for most users. All security would still depend on our security practices (in managing the master key), and only a small handful of crates would even see the gains.
The high opt-in cost has its own advantages: only users who are serious about key management would be likely to do it. However, users already have to trust the security practices of the authors of crates they depend on, and there's not really a way to avoid that.

You can make users handling of keys easy. if you kdf & secretbox them and store them in a user gist or user repository then they will be stored in the cloud, and be encrypted with a strong scheme. They may still be compromised, however, this is no worse than a users laptop or github account being compromised today. There would be a huge benefit though - if you have TUF you have the ability to revoke the keys at a point in history and subsequently heal the user ecosystem by explicitly notifying folks they've fetched potentially unauthorized blobs. This is exactly what TUF is for.

[matthieu-m]

I've been thinking further about the issue of key revocation/rotation, and wondering whether instead of tying the verification to each and every author (a large multitude) it would not be simpler to place one's trust in a handful of trusted entities.

I imagine a lightweight cargo Trust Server whose only role is to interact with a github (or otherwise) repository in which each crate's manifest is signed by this Trust Server own private key.

These Trust Servers would regularly scan existing registries (such as crates.io) and check that the newly published crates have been signed by their(s) author(s). Signed crates manifests are then signed by the Trust Server itself and committed to github.

A simple cargo command lets one add a Trust Server to cargo (it could even come pre-bundled with a known list). When downloading a crate from a registry, cargo can then check the associated public repositories of those servers for the signed manifest of the crate, and require a certain quorum be met.

---

This is slightly more complicated, of course, however it has substantial advantages:

• much fewer dependencies: there are hundreds/thousands/... crate authors, keys are bound to get revoked, and then what?
• redundant dependencies: if one Trust Server disappears, it doesn't matter,
• built-in key rotation: it's easy to add code to the Trust Server to periodically re-sign manifests with a new key, and drop the old ones.

Also, in case a rogue crate appears on registries, the fewer number of Trust Servers makes it still possible to contact their administrators and ask them to unlist the crate in question.

And one day, when we get reproducible builds, we could imagine Advanced Trust Servers rebuilding the crates' binaries and publishing the binaries signatures on their own, providing independent verification that they match the sources.

[tarcieri]

Key revokation is done by deleting the key from GitHub.
When downloading crates, we verify the signature in the registry. We display to the user (published by GitHub user @withoutboats) or something similar, providing them the guarantee that this user did actually publish this data, and the data has not been modified since.

What happens when a user removes a key they used to publish past packages? Do these packages now fail to verify?

How do you handle all of the crates which were never signed in the first place?

This approach seemingly provides AuthN, but not AuthZ: namely, it allows us to check a package has been published by a particular user, but does nothing in terms of providing end-to-end policies for which users are allowed to publish which packages.

To me this is really the power of TUF: packages are claimed by a trusted set of publishers, and after being claimed have end-to-end security from then on.

Its occurred to me that we should also start signing commits to the index. Here's a sketch: We add to the index format's config.json a list of committer keys. Every time we pull, we verify that every commit was signed by one of the keys.

This is really starting to sound a lot like TUF.

(Edit: I do think the index could be potentially leveraged to provide a more lightweight alternative to TUF, but I think doing that well would involve leaning on git for cryptographic properties, which is probably not a wise thing to do until its hash function is upgraded from SHA-1)

[joshtriplett]

I don't think it makes sense to check developer signatures in multiple places. Instead, use a trust model similar to that of Debian: check the developer signature on upload, then sign the index commit with a crates.io key.

[withoutboats]

This feedback seems to misunderstand that the goal of this proposal is to give users assurances without requiring them to trust crates.io. Anything that requires us to trust the security of a private key controlled by crates.io does not meet the goal.

This approach seemingly provides AuthN, but not AuthZ: namely, it allows us to check a package has been published by a particular user, but does nothing in terms of providing end-to-end policies for which users are allowed to publish which packages.

This is accurate but there's no way to provide AuthZ without trusting crates.io services.

[tarcieri]

This is accurate but there's no way to provide AuthZ without trusting crates.io services.

A system which provides AuthN but does not provide AuthZ does not provide useful security properties. You wind up with integrity with no guarantee of authenticity.