Refactor codegen backends in bootstrap #144787

Kobzol · 2025-08-01T16:00:14Z

This PR refactors the codegen backend steps, in preparation to make more progress on the integration of the GCC codegen backend in bootstrap. It does several things:

Splits the CodegenBackend step into two, one for clif and another one for gcc. Even though their code is mostly similar, that's IMO mostly fake similarity, and they do (or will) ultimately require different handling. This was already visible in the requirement of building GCC for cg_gcc, of course.
It is now possible to build both backends (and dist cranelift) even if they are not specified in rust.codegen-backends. It was quite weird that it wasn't possible to even invoke the corresponding codegen backend if the backend wasn't specified in that array, as that array should ideally only change defaults (see later below).
Changes the path specification of these steps to an alias. In other words, instead of compiler/rustc_codegen_cranelift, the step is now built only using rustc_codegen_cranelift or cg_clif. This is done to avoid an annoying clash with x build compiler, which would otherwise build both codegen backends after the 2) change.
Made the copying of codegen backend artifacts more explicit, in particular in the Assemble step.
Codifies the semantics of rust.codegen-backends, which now only affects the defaults of whether a codegen backend will be included in rustc's sysroot and whether it will be disted in x dist by default. We can change the behavior later, e.g. to dist cranelift by default in x dist once it becomes stabilized. Currently I left the existing behavior that we use on CI, I just tried to document it better.

I don't think that this requires a change tracker entry, because the defaults should work the same as before. It is just now possible to do x build/dist rustc_codegen_cranelift even if CLIF is not in the codegen-backends array. It is no longer possible to do ./x build compiler/rustc_codegen_cranelift though, not sure if that requires a change tracker entry.

There is one thing that I didn't touch yet, and that is the fact that rust.codegen-backends not only affects the default behavior of x dist w.r.t. Cranelift, but also of x test. In other words, x test rustc_codegen_cranelift still does not hing if cranelift isn't in rust.codegen-backends. I plan to take a look at this once I get to refactoring the test steps.

r? @jieyouxu

rustbot · 2025-08-01T16:00:19Z

jieyouxu is currently at their maximum review capacity.
They may take a while to respond.

rustbot · 2025-08-01T16:00:22Z

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies bootstrap.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

bootstrap.example.toml

jieyouxu · 2025-08-05T11:52:35Z

@rustbot author

Kobzol · 2025-08-05T14:50:00Z

@rustbot ready

jieyouxu

Thanks, I have some nits/questions

src/bootstrap/src/core/build_steps/compile.rs

src/bootstrap/src/core/build_steps/dist.rs

src/bootstrap/src/core/builder/tests.rs

bootstrap.example.toml

Kobzol · 2025-08-06T12:26:03Z

Pushed review changes.

jieyouxu

Thanks, two nits because I only just noticed

bootstrap.example.toml

src/bootstrap/src/core/build_steps/compile.rs

src/bootstrap/src/core/build_steps/dist.rs

src/bootstrap/src/core/builder/tests.rs

jieyouxu

Thanks

jieyouxu · 2025-08-06T13:44:21Z

@bors r+ rollup=never (build system changes, just in case)

bors · 2025-08-06T13:44:24Z

📌 Commit e89c64f has been approved by jieyouxu

It is now in the queue for this repository.

Zalathar · 2025-08-07T07:38:01Z

Let's run a never PR while I investigate the recent rollup failure.

@bors p=1

Zalathar · 2025-08-07T08:05:53Z

Undoing my previous priority bump, since the immediate scheduling concern was solved.

@bors p=0

lqd · 2025-08-08T17:08:30Z

I knew it, we may need to close the tree

Kobzol · 2025-08-08T17:12:10Z

@/bors treeclosed=100 A test is failing on CI.

I guess we just comment out the test for now?

Revert #143906 This reverts commit 71f0469, reversing changes made to 995ca3e. Reverts #143906, which was merged in #145043. It seems like it is causing test failures on CI that block merges (#144787 (comment). try-job: x86_64-gnu-aux

Kobzol · 2025-08-08T20:52:25Z

#145116 landed, so reopening the tree.

@/bors treeclosed-

Kobzol · 2025-08-08T20:53:02Z

@bors retry

Revert #143906 This reverts commit 71f04692c32e181ab566c01942f1418dec8662d4, reversing changes made to 995ca3e532b48b689567533e6b736675e38b741e. Reverts rust-lang/rust#143906, which was merged in rust-lang/rust#145043. It seems like it is causing test failures on CI that block merges (rust-lang/rust#144787 (comment). try-job: x86_64-gnu-aux

bors · 2025-08-10T02:32:45Z

⌛ Testing commit 1a2ee80 with merge 915a766...

bors · 2025-08-10T05:38:55Z

☀️ Test successful - checks-actions
Approved by: jieyouxu
Pushing 915a766 to master...

github-actions · 2025-08-10T05:42:09Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 41ede7b (parent) -> 915a766 (this PR)

Test differences

Show 3 test diffs

Stage 0

core::builder::tests::snapshot::dist_cranelift_by_default: [missing] -> pass (J0)

Additionally, 2 doctest diffs were found. These are ignored, as they are noisy.

Job group index

J0: aarch64-apple, aarch64-gnu, aarch64-gnu-llvm-19-2, aarch64-msvc-2, i686-gnu-2, i686-gnu-nopt-2, i686-msvc-2, x86_64-apple-1, x86_64-gnu, x86_64-gnu-llvm-19-1, x86_64-gnu-llvm-19-3, x86_64-gnu-llvm-20-1, x86_64-gnu-llvm-20-3, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-2, x86_64-msvc-2

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 915a766b2f9fd53a8cd7b1fad003d3f8e488ff4b --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

dist-aarch64-apple: 5653.7s -> 6853.5s (21.2%)
aarch64-apple: 6505.5s -> 5195.7s (-20.1%)
dist-apple-various: 3785.8s -> 4428.3s (17.0%)
x86_64-apple-2: 5102.1s -> 5874.6s (15.1%)
dist-loongarch64-musl: 4959.9s -> 5497.4s (10.8%)
x86_64-mingw-2: 7602.0s -> 8335.7s (9.7%)
x86_64-apple-1: 6599.6s -> 7223.5s (9.5%)
aarch64-gnu-llvm-19-2: 2650.6s -> 2404.4s (-9.3%)
tidy: 104.3s -> 113.6s (8.9%)
dist-riscv64-linux: 4767.1s -> 5090.6s (6.8%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2025-08-10T07:43:26Z

Finished benchmarking commit (915a766): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 2.3%, secondary -2.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.3%	[1.1%, 3.5%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.3%	[-2.3%, -2.3%]	1
All ❌✅ (primary)	2.3%	[1.1%, 3.5%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 463.257s -> 463.478s (0.05%)
Artifact size: 377.39 MiB -> 377.38 MiB (-0.00%)

rustbot assigned jieyouxu Aug 1, 2025

Kobzol changed the title ~~Refactor codegen backends~~ Refactor codegen backends in bootstrap Aug 1, 2025

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Aug 1, 2025

This comment has been minimized.

Sign in to view

GuillaumeGomez reviewed Aug 3, 2025

View reviewed changes

bootstrap.example.toml Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 5, 2025

Kobzol force-pushed the codegen-backend-restructure branch from 588f96f to 0a3b2d7 Compare August 5, 2025 14:49

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 5, 2025

jieyouxu requested changes Aug 6, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 6, 2025

jieyouxu approved these changes Aug 6, 2025

View reviewed changes

bootstrap.example.toml Show resolved Hide resolved

src/bootstrap/src/core/build_steps/compile.rs Show resolved Hide resolved

src/bootstrap/src/core/build_steps/dist.rs Show resolved Hide resolved

src/bootstrap/src/core/builder/tests.rs Show resolved Hide resolved

Kobzol force-pushed the codegen-backend-restructure branch from 4235812 to e89c64f Compare August 6, 2025 13:35

jieyouxu approved these changes Aug 6, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 6, 2025

Shourya742 approved these changes Aug 7, 2025

View reviewed changes

This was referenced Aug 8, 2025

add a scope for if let guard temporaries and bindings #143376

Merged

Miri: non-deterministic floating point operations in foreign_items #143906

Merged

Revert #143906 #145116

Merged

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 8, 2025

bors added the merged-by-bors This PR was explicitly merged by bors. label Aug 10, 2025

bors merged commit 915a766 into rust-lang:master Aug 10, 2025
11 checks passed

rustbot added this to the 1.91.0 milestone Aug 10, 2025

Kobzol deleted the codegen-backend-restructure branch August 10, 2025 06:08

matthiaskrgr mentioned this pull request Aug 10, 2025

cranelift: librustc_driver-2bb05b679736573e.so: cannot open shared object file #145201

Open

Kobzol mentioned this pull request Aug 10, 2025

Ship correct Cranelift library in its dist component #145207

Open

Refactor codegen backends in bootstrap #144787

Refactor codegen backends in bootstrap #144787

Conversation

Kobzol commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Aug 1, 2025

Uh oh!

rustbot commented Aug 1, 2025

Uh oh!

This comment has been minimized.

Uh oh!

jieyouxu commented Aug 5, 2025

Uh oh!

Kobzol commented Aug 5, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kobzol commented Aug 6, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

jieyouxu commented Aug 6, 2025

Uh oh!

bors commented Aug 6, 2025

Uh oh!

Zalathar commented Aug 7, 2025

Uh oh!

Zalathar commented Aug 7, 2025

Uh oh!

lqd commented Aug 8, 2025

Uh oh!

Kobzol commented Aug 8, 2025 • edited by jieyouxu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Aug 8, 2025 • edited by jieyouxu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Aug 8, 2025

Uh oh!

bors commented Aug 10, 2025

Uh oh!

bors commented Aug 10, 2025

Uh oh!

Uh oh!

github-actions bot commented Aug 10, 2025

Test differences

Stage 0

Job duration changes

Uh oh!

rust-timer commented Aug 10, 2025

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Uh oh!

Kobzol commented Aug 1, 2025 •

edited

Loading

Kobzol commented Aug 8, 2025 •

edited by jieyouxu

Loading

Kobzol commented Aug 8, 2025 •

edited by jieyouxu

Loading