Implement release process for rustup #84

jdno · 2024-05-23T13:44:56Z

New versions of rustup are currently released manually by running the sync-dist.py script in the rust-lang/rustup repository multiple times. This process should be automated so that it reproducible and less error-prone. We also want to add more sanity checks in the future, which are more convenient to implement in Rust rather than in Python.

In rust-lang/rustup#3819, we discussed reimplementing the script as part of promote-release. This pull request makes a first attempt at migrating the existing functionality into this repository.

jdno · 2024-05-23T13:45:28Z

The following features still need to be implemented:

Smoke tests for the rustup release process (similar to run.sh for releases)
Cache invalidations in the CDNs

jdno

I'm not very happy with this implementation. It feels like we're trying to force a square through a round hole. Most of the configuration for promote-release is not needed for rustup, doesn't map 1:1 to its release process (e.g. channels), and passing in the version dynamically as an argument is difficult. 🙁

jdno · 2024-05-23T13:46:23Z

src/rustup.rs

+    /// [rust-lang/rustup]: https://github.com/rust-lang/rustup
+    pub fn promote_rustup(&mut self) -> anyhow::Result<()> {
+        println!("Checking channel...");
+        if self.config.channel != Channel::Stable && self.config.channel != Channel::Beta {


This feels really weird, since the current environments for rustup are dev-static and prod.

Why are we making the channel be the condition here? Promote release already has dev static and static, as two separate environments - should be able to ignore channels I'd expect. (In a manner similar to promote branches ignoring them iirc)

Without channels, how would you distinguish between beta and stable releases? Only be setting the respective environment variables (e.g. UPLOAD_BUCKET)?

I also think we should always have the channel be stable and then do the prod/dev distinction like Rust does.

Without channels, how would you distinguish between beta and stable releases?

dev and prod releases would happen in different CodeBuild pipelines, which have different configured secrets/env vars.

PS: The status quo is that we are calling dev-static our beta environment, and probably that's what has caused the confusion (because it's not a proper "channel" like with the releases of Rust itself)? In the end I wouldn't refuse getting something closer to Rust if that helps simplify your workflow.

This still needs to be addressed 🙂

jdno · 2024-05-23T13:49:12Z

src/rustup.rs

+        let version = self
+            .current_version
+            .as_ref()
+            .ok_or_else(|| anyhow!("failed to get current version for rustup release"))?;


This doesn't work, since self.current_version is None when this runs. Apparently, the version is set much later in the normal release process.

The easiest workaround might be setting a new PROMOTE_RELEASE_RUSTUP_VERSION environment variable, but that would need to be updated manually before running the CodeBuild project. That feels risky, since it's very easy to forget this step and right now we would just override existing artifacts. 😬

I think we should figure out how to persist the intended version with the artifacts. That should be possible similarly to how rust has a version containing file checked in, and we can upload that directly to the S3 bucket in rustup's existing "CI".

I agree with Mark that the best approach here is for rustup to store the version number in a file in the repository, so that they can update it on their own.

In general, regarding arguments, I don't see any problem with sticking with environment variables. We never invoke the CodeBuild job directly, we always go through https://github.com/rust-lang/simpleinfra/blob/master/release-scripts/promote-release.py, which has a CLI parser and then sets the environment variables.

An early step in our release process is a version bump in Cargo.toml(e.g. rust-lang/rustup@cfca13c). Is that useful for this particular purpose?

The version is now read from the Cargo.toml in the latest commit on the stable branch.

@djc Now that we're at it, maybe it makes sense to use workspace-wide version numbers? I don't see how rustup-download can be at a different version from rustup itself.

@jdno FYI: ~~rust-lang/rustup#4041~~ rust-lang/rustup#4061 now uses a workspace-wide version number.

Really sorry for the back and forth, but since rust-lang/rustup#4277 rustup is now a single crate, so the workspace-wide version number no longer applies.

jdno · 2024-05-23T13:55:50Z

One approach might be to refactor the interface for this tool and turn it into a CLI with clap. That way, each command can have its own configuration and parameters. We can still get the arguments from the environment, but it would be much clearer what configuration is needed by each command.

Mark-Simulacrum · 2024-05-23T15:55:04Z

One approach might be to refactor the interface for this tool and turn it into a CLI with clap. That way, each command can have its own configuration and parameters. We can still get the arguments from the environment, but it would be much clearer what configuration is needed by each command.

I'm not sure CLI arguments really make any strong difference there? Whether it's environment variables or flags shouldn't really matter - we can add a common prefix if we want, but the two feel very similar to me. (Modulo --help but that is useful more to humans and it's not intended that humans are running this stuff).

pietroalbini · 2024-05-23T16:15:10Z

src/rustup.rs

+    /// `rustup` uses different branches to manage releases. Whenever a commit is pushed to the
+    /// `stable` branch in [rust-lang/rustup], GitHub Actions workflows build release artifacts and
+    /// copy them into `s3://dev-static-rust-lang-org/rustup/dist/`.


This is not great, I'd prefer if we tweak rustup's CI to upload them to a location like s3://rustup-artifacts/${commit}, check what is the latest commit hash on the stable branch, and download from that location. Every build overriding the previous build feels iffy.

The bucket has been created here and a PR is open here to upload artifacts to s3://rustup-builds/builds/${commit}/.

jdno · 2024-05-28T11:42:36Z

@Mark-Simulacrum @pietroalbini Thanks for the feedback! I've drafted a new release process with the suggested changes in rust-lang/rustup#3844. If that makes sense I'm gonna implement the changes here.

rami3l · 2024-07-10T05:12:33Z

I've made rust-lang/rustup#3932 so that we can share our notes on how to improve Rustup's release process.

Currently beta and stable releases are available, but I think master builds would be interesting as well for dogfooding and/or quickly verifying recent fixes.

The override feature for the rustup version has been removed, since the version seems to be hardcoded in the binary as well. So it was possible to change the path where the artifacts were stored in S3, but using the version to verify that rustup got updated wasn't possible.

Comparing the Rustup versions before and after the update does not work, since the version in the GitHub repository is not bumped after a release. We're going back to the override version so that we can force rustup to self update, which we can check for without comparing version numbers. This reverts commit c9b96fe.

After publishing a new rustup release, we are now invalidating both CloudFront and Fastly. This fixes [rust-lang/simpleinfra#415]. [rust-lang/simpleinfra#415]: rust-lang/simpleinfra#415

jdno · 2025-03-31T13:33:08Z

@Mark-Simulacrum @pietroalbini @rami3l I think this is in a good state now for an actual review. The last item on my mental todo list is checking that the documentation here and in rust-lang/rustup#3844 actually match the implementation.

pietroalbini · 2025-03-31T19:29:29Z

Changes look good to me! Just left a comment on #84 (comment).

local/Dockerfile

+    rm awscliv2.zip; \
+    ./aws/install

 # Install rustup while removing the pre-installed stable toolchain.


rami3l · 2025-06-30T05:43:58Z

@jdno Sorry for the interruption, but I'm wondering what is blocking this PR from being merged since a few months ago. If there is anything I can help, please let me know! 🙏

jdno · 2025-07-14T11:58:40Z

No worries and sorry for the delay on this...

There is one more item of development work that I'm tracking, and that is changing the parsing of the Cargo.toml from a workspace to a crate again. After that, it's about testing the implementation and that's where I got stuck since I just don't have the context and didn't have the time to look into it.

My suggestion at this point is that I update the implementation in this PR, and then hand the testing of the implementation over to either you guys or the release team. I think the two of you can iterate more quickly on the final details than I can as a middleman between the two teams...

rami3l · 2025-07-14T13:15:24Z

@jdno Sounds good to me! I'm just wondering if there has been any prior art in terms of testing this properly or we kind of have to figure out on the go. I'm asking because I'm actually planning for another release soon™️... If you would like us to do some shadow testing or anything I guess we can arrange that.

pietroalbini

Left a few comments.

src/rustup.rs

rami3l · 2025-10-16T10:43:46Z

@pietroalbini What can I (and the whole t-rustup) do to push this forward? I'd like to try it out in future releases :)

jdno · 2025-10-16T12:19:44Z

I saw the conflicts a while ago, but didn't have time to pick this back up due to RustConf and all the handover work before leaving the Foundation. 🙈

Here are the tasks that I'm still tracking to land this:

Get this PR approved by t-release
Review the documentation in Document the new release process for Rustup rustup#3844 and ensure consistent with the implementation
Modify the start-release Lambda to support the new command
Grant the rustup team permission to start a new release
Test the implementation with a beta release

And then there are a few small cleanup tasks that are tracked in rust-lang/simpleinfra#420.

I haven't been following the work that the release team did with the start-release Lambda and didn't get a clear answer about their preferences for rolling this out. I guess if we can get the PR unblocked and merged, we can go through the rest of the other tasks without the release team. 🤷‍♂️

Mark-Simulacrum · 2025-10-16T23:03:35Z

I think I replied somewhere, but maybe I just dreamed that up - release team ideally isn't involved at all, and the start-release stuff should either grant rustup team access to kickoff rustup releases or be duplicated to do so. We want to give rustup independence in issuing releases of rustup.

rami3l · 2025-10-17T05:45:40Z

@Mark-Simulacrum I think @rust-lang/rustup is okay with that change in responsibility; however I'm wondering would your comment indicate a change in the plan moving forward?

djc · 2025-10-17T07:16:40Z

Also, I don't think t-rustup necessarily wants to take responsibility for promote-release. While we could certainly contribute minor changes that rustup needs, I don't think I'm necsesarily the best person to review/finalize this PR.

Mark-Simulacrum · 2025-10-17T11:03:58Z

I'd expect infra to continue to largely own this repository. The only delta is replacing step 1 above with infra approval.

rami3l · 2025-10-17T13:22:26Z

Indeed, I didn't expect t-rustup to take responsibility for the release infrastructure part of the thing, if I haven't been clear enough.

That said, would it be fine for @rust-lang/infra to now review and potentially merge this patch?

jdno added 3 commits May 20, 2024 12:56

Fix error message when parsing actions

00e1a90

Create action for rustup

0a64d25

Implement release process for rustup

8d4a447

jdno commented May 23, 2024

View reviewed changes

pietroalbini reviewed May 23, 2024

View reviewed changes

jdno mentioned this pull request May 28, 2024

Purge Fastly cache after release rust-lang/rustup#3819

Closed

jdno mentioned this pull request May 31, 2024

Tracking issue: Refactor release process for Rustup rust-lang/simpleinfra#420

Open

16 tasks

jdno added 6 commits June 27, 2024 16:12

Extract channel check for Rustup into function

d40ff45

Get latest commit from Rustup's stable branch

b803170

Fetch next Rustup version from GitHub

da3164d

Download Rustup artifacts for a given commit

49980a3

Pass Rustup version to archive and manifest

c4cff77

Update documentation for Rustup release process

7ccff04

jdno added 12 commits September 24, 2024 12:25

Refactor run script to execute different local releases

ede844c

Support other platforms than Linux

1e516af

Download Rustup files from CDN

b1dd2a2

Build promote-release inside the container if necessary

b843e51

Get download path from config

f5c3c5e

Fix upload directory

a947033

Get version from user-provided commit

883b5ed

Don't require GitHub credentials to fetch Rustup metadata

ad86e13

Create rustup-builds bucket in local environment

694e3a4

Fix upload destination for Rustup archives

8132d69

Fix upload destination for stable Rustup release

89f7324

Fix upload destination for Rustup manifest

f82d531

jdno added 2 commits March 25, 2025 15:29

Fix archiving of rustup artifacts

af7e298

jdno force-pushed the promote-rustup branch from 08d092c to c9b96fe Compare March 25, 2025 15:13

jdno added 4 commits March 25, 2025 16:27

Invalidate CDNs when publishing new rustup release

9a096ec

After publishing a new rustup release, we are now invalidating both CloudFront and Fastly. This fixes [rust-lang/simpleinfra#415]. [rust-lang/simpleinfra#415]: rust-lang/simpleinfra#415

Fix auto-formatting in GitHub Action

acb0303

Skip CloudFront validations in loca rustup tests

1edda59

jdno marked this pull request as ready for review March 31, 2025 13:32

rami3l reviewed Apr 1, 2025

View reviewed changes

local/Dockerfile Outdated

rm awscliv2.zip; \

./aws/install

# Install rustup while removing the pre-installed stable toolchain.

This comment was marked as resolved.

Sign in to view

rami3l mentioned this pull request Jul 22, 2025

1.29.0 Release Planning rust-lang/rustup#4423

Open

4 tasks

jdno added 5 commits July 23, 2025 15:06

Merge branch 'master' into promote-rustup

fa4be8e

Update return types after recent refactoring

08d6c8c

Revert back to single crate

9e85028

Update obsolete comment

194e36c

Fix new clippy warnings

69f572d

pietroalbini reviewed Jul 23, 2025

View reviewed changes

src/rustup.rs Show resolved Hide resolved

src/rustup.rs Outdated Show resolved Hide resolved

Call function to promote artifacts to S3

fea7569

rami3l mentioned this pull request Oct 18, 2025

Add vs-components.toml rust-lang/rustup#4543

Draft

Implement release process for rustup #84

Are you sure you want to change the base?

Implement release process for rustup #84

Uh oh!

Conversation

jdno commented May 23, 2024

Uh oh!

jdno commented May 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdno left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rami3l Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rami3l Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rami3l Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdno commented May 23, 2024

Uh oh!

Mark-Simulacrum commented May 23, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdno Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdno commented May 28, 2024

Uh oh!

rami3l commented Jul 10, 2024

Uh oh!

jdno commented Mar 31, 2025

Uh oh!

pietroalbini commented Mar 31, 2025

Uh oh!

This comment was marked as resolved.

Uh oh!

rami3l commented Jun 30, 2025

Uh oh!

jdno commented Jul 14, 2025

Uh oh!

rami3l commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pietroalbini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rami3l commented Oct 16, 2025

jdno commented May 23, 2024 •

edited

Loading

rami3l Apr 1, 2025 •

edited

Loading

rami3l Oct 9, 2024 •

edited

Loading

rami3l Apr 7, 2025 •

edited

Loading

jdno Jun 27, 2024 •

edited

Loading

rami3l commented Jul 14, 2025 •

edited

Loading