perf: speed up RTS testing on macOS — split #rts build/check + parallelise test variants

[ggreif]

Summary

Two complementary attacks on RTS test wall-clock — parallelisation makes the test suite finish faster when it does run, and the new build/check split lets nix build .#rts skip running it altogether on Mac (where it has no cache hit).

Build-only .#rts and check-running .#rts-checked (latest)

nix/rts.nix factored into two derivations sharing all build attrs:

• rts-build — doCheck = false, produces the cross-compiled wasm artifacts only.
• rts-checked — doCheck = true, additionally runs make -j8 test.

Wired into flake.nix so:

• .#rts-checked is always the test-running variant.
• .#rts aliases rts-checked on Linux (Hydra-cached, fast — no behaviour change) and rts-build on darwin (skips the slow path).
• nightly-macos-test gains an explicit rts-checked job that builds .#rts-checked directly, so the darwin-side cargo test suite still runs daily on master.

Why: on macos-latest (aarch64-darwin) the host-side cargo test is a from-source build with no Hydra cache hit (combo not in cachix), adding ~10+ min to every artifact build. On Linux it's a fast cache hit. The split is platform-asymmetric on purpose: PR-blocking on Linux, scheduled-only on darwin.

Net effect: nix build .#rts terminates fast on the Mac everybody develops on; PR CI coverage on Linux is unchanged; darwin RTS test coverage moves from "every PR (slow)" to "nightly".

Phase A: Variant-level parallelism

• Separate CARGO_TARGET_DIR per variant (target-<name>) to avoid cargo lock contention.
• define/eval Makefile template generates build + per-module run targets.
• make -j8 test in nix checkPhase.

Phase B: Per-module parallelism via wasmtime --invoke

• Each test module gets a #[no_mangle] pub extern "C" fn test_<mod>() entry point.
• Makefile runs wasmtime --invoke test_<mod> per module — works on wasm64-unknown-unknown without WASI.
• 3 variants × 24 modules = 72 parallel targets.

Phase C: GC seed chunking

• Split 100 GC random seeds into 10 chunks of 10 (test_gc_chunk_0..9).
• Separate gc_predefined (hand-crafted heaps) from gc_components (incremental/compacting internals).
• Split persistence into persistence_small (up to 10k objects) and persistence_20k (20k objects).
• Heavy tests ordered first in TEST_MODULES so make -j starts long poles early.

Dynamic test seeds

• Stabilization small tests use a seed derived from git rev-parse HEAD at build time.
• Each commit exercises different random heap configurations.
• Fallback to fixed seed 4711 when not in a git repo (nix sandbox).
• The heavy persistence_20k test uses fixed seed 4711 for predictable CI runtime.

Bug fix: heap size scaling

• heap_size_for_gc for incremental GC ignored total_heap_size_bytes, always returning 3 * PARTITION_SIZE (192 MB).
• For seeds generating large object graphs, the dynamic heap exceeded this fixed size.
• Fix: max(3 * PARTITION_SIZE, 2 * total_heap_size_bytes).
• Discovered via seed 20_000 which generates a dense 20k-object graph.

Other improvements

• test -f guard before wasmtime to fail fast if cargo didn't produce the binary.
• WASMTIME_BACKTRACE_DETAILS=1 for better crash diagnostics.
• Removed unnecessary unsafe blocks in test entry points.

Observed speedup (parallelisation alone): from 2+ hours sequential to ~90 minutes parallel on macOS, limited by persistence_20k (Amdahl's law). The .#rts build-skip on top of that cuts the Mac path further whenever tests aren't needed (artifacts, dev shells, anything not gating on RTS unit tests).

Test plan

• CI green
• nix build .#rts on darwin skips cargo test (timing check)
• nix build .#rts-checked on darwin runs and passes cargo test
• nix build .#rts on Linux still runs cargo test (regression guard)
• Next nightly nightly-macos-test run shows the new rts-checked job — it does

🤖 Generated with Claude Code

[github-actions]

Comparing from 6a81fc5 to d8c9c87:
The produced WebAssembly code seems to be completely unchanged.
In terms of gas, no changes are observed in 5 tests.
In terms of size, no changes are observed in 5 tests.

[alexandru-uta]

What are the added benefits? Is it really faster, how much?

[ggreif]

What are the added benefits? Is it really faster, how much?

It is still dominated by the 20000-tree, but at least this one now runs in parallel to the others. I was fed up with the slowness on the Mac, so this might help. But I haven't done A/B testing yet.

The other thing is that this introduces different rand seeds per 10000-tree. The fixed seed is kept for the big one for less surprises in run time.

[ggreif]

Keeping this as draft, as I am brainstorming how the bottleneck can be improved.

EDIT: The bottleneck is resolved by disabling the checkPhase for MacOS and shifting it to a nightly CI task. A proper night-shift, so to say :-)