Skip to content

Improve devenv shell and task warm-start performance#464

Open
schickling-assistant wants to merge 15 commits intomainfrom
schickling/2026-03-26-devenv-perf
Open

Improve devenv shell and task warm-start performance#464
schickling-assistant wants to merge 15 commits intomainfrom
schickling/2026-03-26-devenv-perf

Conversation

@schickling-assistant
Copy link
Copy Markdown
Collaborator

@schickling-assistant schickling-assistant commented Mar 26, 2026

Summary

  • add an outer setup fingerprint cache so warm devenv shell can skip unchanged bootstrap work
  • keep initial workspace normalization on shell entry via mr:apply, while removing ts:emit from shell entry
  • speed up warm task status paths for pnpm:install, genie:run, and megarepo tasks, and filter noEmit projects out of the ts:emit graph

Benchmarks

All timings below are local wall-clock measurements taken during this PR. Means are arithmetic means of the listed samples.

Warm Shell Progression

Scenario Samples Mean % change vs baseline Notes
Baseline before repo-side fixes 7.93s, 8.46s, 8.56s 8.32s 0.0% Warm devenv shell --no-tui true before removing heavy shell-entry work
After removing only ts:emit from shell entry 4.45s, 4.55s, 4.58s 4.53s -45.6% Isolates ts:emit as a major shell-entry bottleneck
Counterfactual with all shell-entry setup disabled 0.92s, 0.94s, 1.10s 0.99s -88.1% Shows the remaining floor once repo setup is out of the way
Previous branch head 87b57eb24 steady state 2.20s, 2.23s 2.22s -73.3% Warm devenv shell --no-tui true after the cache and task-status changes before the setup-fingerprint rewrite
Current head 797bbd1c7 first run after setup-fingerprint invalidation on pinned devenv 2.0.7 19.86s 19.86s +138.7% Refresh-like run that reseeds derived shell state before the next warm entry
Current head 797bbd1c7 steady state on pinned devenv 2.0.7 2.33s, 2.16s 2.25s -73.0% Current supported warm-shell path; stays under the <3s target
Current head 797bbd1c7 steady state on local devenv 2.0.6+55d2bb4 4.97s 4.97s -40.3% Older local CLI still misses the target even with the repo-side fixes

Notes:

  • Earlier on this branch, devenv shell --no-eval-cache --no-tui true was about 96s, which remains a separate cold-path / eval problem outside this PR's warm-path scope.
  • The pinned devenv 2.0.7 runtime is materially faster on the final warm-shell path than the older ambient 2.0.6 CLI.

Setup Fingerprint Bottleneck

Metric Measurements Mean % change vs direct-gate baseline Interpretation
Direct setup fingerprint before tool-identity rewrite 1.67s, 1.24s, 1.15s 1.35s 0.0% Setup invalidation used pnpm --version, genie --version, and mr --version, so shell entry paid each CLI startup cost
Direct setup fingerprint after resolved-tool rewrite 0.53s, 0.53s, 0.56s 0.54s -60.1% Keeps tool-change invalidation by hashing mutable resolved tool targets instead of spawning the CLIs
setup:gate inside pinned devenv 2.0.7 warm shell 382.28ms, 393.97ms 388.13ms -71.4% vs old direct-gate baseline Confirms the cheaper fingerprint carries through into steady-state shell entry

Comparable Warm Task Progression

Task Earlier baseline before fixes Current refresh run Current steady state % change vs early baseline Notes
devenv tasks run pnpm:install --mode single --no-tui 4.79s 18.26s 2.17s, 2.21s, 2.23s, 2.27s, 2.44s -52.8% steady-state Main warm status bottleneck addressed in this PR
devenv tasks run genie:run --mode single --no-tui 3.05s 4.41s 0.73s, 0.74s -75.7% steady-state First run seeds or refreshes cache state; steady state is the relevant warm number
devenv tasks run ts:emit --mode single --no-tui 5.10s 7.05s 0.40s, 0.44s -91.8% steady-state Explicit task only; removed from shell entry

Warm Task State On 87b57eb24

Task Refresh run Steady-state samples Steady-state mean % change refresh -> steady Notes
devenv tasks run mr:apply --mode single --no-tui 4.34s 0.70s, 0.71s 0.71s -83.6% Current bootstrap task; no exact historical row because shell bootstrap used mr:sync earlier
devenv tasks run genie:run --mode single --no-tui 4.41s 0.73s, 0.74s 0.74s -83.2% Included here for current-state completeness
devenv tasks run pnpm:install --mode single --no-tui 18.26s 2.17s, 2.21s, 2.23s, 2.27s, 2.44s 2.26s -87.6% Warm status still dominates the remaining repo-owned shell-entry work before the final setup-fingerprint rewrite
devenv tasks run ts:emit --mode single --no-tui 7.05s 0.40s, 0.44s 0.42s -94.0% Explicit task only

pnpm:install Status Bottleneck

Metric Measurements % change Interpretation
OTEL child span before the latest projection-hash rewrite 1397ms, 2113ms pre-rewrite context Trace-backed attribution that pnpm:install:status was the dominant warm-shell child span
Isolated compute_workspace_state_hash 0.12s to 0.14s context only Manifest hashing was never the main problem
Isolated compute_install_state_hash 0.47s to 0.50s context only Includes pnpm --version, workspace hash, and GVS links root
Isolated structural projection fingerprint before rewrite 1.68s, 1.70s, 1.72s, 1.78s, 1.90s baseline This shell pipeline was the main pnpm warm-status bottleneck
Isolated structural projection fingerprint after rewrite 0.08s, 0.08s, 0.08s, 0.09s, 0.09s -95.1% About 20x faster with the same hash semantics via a one-process Node helper
Isolated light structural hash (dirs + .modules.yaml) 0.04s -97.7% vs old full fingerprint Faster, but rejected because it misses removed / retargeted projected symlinks
Isolated broken-symlink scan 0.26s to 0.27s -84.8% vs old full fingerprint Useful signal, but too weak on its own

pnpm Fault Matrix

The synthetic fault matrix used to validate footguns looked like this:

Corruption case Full structural fingerprint Light hash Broken-link scan Semantic health check
Missing package node_modules dir catches catches misses misses
Missing projected package symlink catches misses misses misses
Retargeted projected package symlink catches misses misses misses
Broken projected symlink target catches misses catches misses
Missing transitive dependency projection misses misses misses catches

This is why the PR keeps the current structural warm-status contract and only optimizes its implementation, instead of switching to the lighter hash or to the deeper semantic health check.

Blockers Before Merge

This branch is intentionally strict. The downstream migration needs to land before #464 can merge.

  • Migrate schickling/megarepo-all
    • schickling/gvs-downstream-cleanup: replace megarepo:* task references with mr:*, pass pnpm explicitly to mk-pnpm-cli, refresh tools/alignment-cli/nix/build.nix pnpmDepsHash, and repin repos/effect-utils / workflow-genie imports away from main
    • schickling/fix-alignment-summary: replace megarepo:* task references with mr:*, pass pnpm explicitly to mk-pnpm-cli, refresh tools/alignment-cli/nix/build.nix pnpmDepsHash, and repin repos/effect-utils / workflow-genie imports away from main
    • schickling/2026-03-25-pnpm-11: replace megarepo:* task references with mr:*, pass pnpm explicitly to mk-pnpm-cli, refresh tools/alignment-cli/nix/build.nix pnpmDepsHash, and repin repos/effect-utils / workflow-genie imports away from main
    • schickling/2026-03-17-link-protocol-issue: replace megarepo:* task references with mr:*, pass pnpm explicitly to mk-pnpm-cli, refresh its branch-specific tools/alignment-cli/nix/build.nix pnpmDepsHash, and repin repos/effect-utils / workflow-genie imports away from main

Validation

  • bash nix/devenv-modules/tasks/shared/tests/setup-cache.test.sh
  • CI=1 bash nix/devenv-modules/tasks/shared/tests/pnpm-task-smoke.test.sh
  • OTEL_MODE=local CI=1 devenv tasks run nix:test --mode before --no-tui
  • OTEL_MODE=local CI=1 devenv tasks run check:all --mode before --no-tui
  • local megarepo-all validation found the strict downstream migration blockers above; they are intentionally not shimmed in this branch

Tradeoffs

  • mr:apply still runs on initial shell bootstrap so the worktree is normalized, but warm shells now skip it when the outer setup fingerprint and inner task caches are unchanged
  • ts:emit remains explicit rather than shell-entry work because its correct no-op detection still depends on TypeScript's build planner and is materially slower than the other setup tasks
  • the pnpm warm path still uses a structural projection fingerprint rather than the deeper semantic health check; this preserves the existing skip contract while avoiding the shell pipeline overhead that made the status path expensive

Acting on behalf of the user.

  • Add continuous devenv perf regression measurement for cold/warm shell and key task paths so we catch startup regressions early.

@github-actions github-actions Bot requested a review from schickling March 26, 2026 21:39
@schickling
Copy link
Copy Markdown
Collaborator

@codex

@schickling schickling force-pushed the schickling/2026-03-26-devenv-perf branch from 21e8c79 to a71e01e Compare April 1, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants