Test Suite Execution Optimizer with Adaptive Fast-Path Sampling

[github-actions]

Strategic Improvement

Faster test feedback reduces wasted build loop iterations and context exhaustion. Build an optimizer that tracks test execution time and failure rates, runs fast tests first, samples slow tests adaptively, and learns which tests fail most frequently. Integrate with build loop to provide sub-30-second smoke test results before running full suite. Reduces cost (P3) and improves reliability (P0) simultaneously.

Acceptance Criteria

• Track per-test execution time and failure rate in .claude/test-stats.json
• Generate test execution plan: fast tests first, then stratified sample of slow tests
• Smoke test mode runs top-20 fastest tests + top-10 historically-failing tests
• Full test mode runs all tests but in optimized order (likely-to-fail first)
• Integration with sw-loop.sh --fast-test-cmd for automatic smoke test usage
• Dashboard showing test execution time distribution and failure hotspots
• Measure and report feedback time improvement (target: <30s for smoke tests)

Context

• Priority: P0
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate (via faster feedback)

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Strategic improvement to build loop efficiency with clear acceptance criteria, measurable benefits, and multi-subsystem integration (test tracking, loop integration, dashboard); standard scope fits existing infrastructure patterns.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24282384112

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (17 lines)

The implementation plan has been written to .claude/plan.md. Here's a summary of the approach:

Strategy: Extend the existing scripts/lib/test-optimizer.sh library (which is already sourced by sw-loop.sh but unused) rather than building from scratch. The foundation is 90% in place.

Key deliverables:

1. 6 new functions in the existing library: testopt_generate_stats, testopt_smoke_plan, testopt_full_plan, testopt_adaptive_sample, testopt_update_stats, testopt_prune_history
2. New CLI script scripts/sw-test-optimizer.sh with stats/plan/smoke/dashboard subcommands
3. sw-loop.sh integration: Auto-generate smoke commands, record test history after each execution
4. Stats persistence in .claude/test-stats.json with per-test timing and failure rates

Critical design decisions:

• Smoke plan: top-20 fastest + top-10 highest-failing tests (deduplicated), targeting <30s
• New tests with no history get a default fail_rate of 0.5 to ensure they're included in smoke runs (addresses the P0 reliability concern)
• Atomic writes throughout, history pruning to prevent unbounded growth
• Stratified adaptive sampling: all fast tests, 50% medium, 25% slow (by fail_rate)

Files touched: 5 modified (lib, loop, CLI router, package.json, test suite) + 1 new (CLI script). Minimal blast radius.

---

Generated by shipwright pipeline at 2026-04-11T12:33:51Z

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (16 lines)

The ADR has been written to .claude/design.md. Here's a summary of the key architectural decisions:

Approach: Extend the existing scripts/lib/test-optimizer.sh (already sourced but dormant in sw-loop.sh) with 6 new functions rather than building from scratch. The foundation is 90% in place.

5 critical design decisions documented:

1. D1: Unknown tests get fail_rate=0.5 to prevent silent regression gaps in smoke plans
2. D2: Split storage -- repo-scoped stats in .claude/test-stats.json, user-scoped history in ~/.shipwright/optimization/
3. D3: Hybrid smoke plan (top-20 fastest + top-10 highest-failing, deduplicated) targeting <30s
4. D4: Stratified adaptive sampling (100% fast / 50% medium / 25% slow by fail_rate)
5. D5: History pruning at 100 entries per test + 5MB hard cap

4 alternatives rejected with specific rationale: TypeScript optimizer (architectural outlier), SQLite history (unjustified complexity), standalone daemon (premature abstraction), pure fail-rate smoke (violates <30s target).

Blast radius: 5 files modified + 1 new file. The sw-loop.sh integration (~25 lines) is guarded behind function existence checks so the optimizer being absent is a silent no-op.

14 validation criteria cover the smoke plan algorithm, cold start behavior, atomic writes, CLI integration, backward compatibility with --fast-test-cmd, and Bash 3.2 compliance.

---

Generated by shipwright pipeline design stage at 2026-04-11T12:41:10Z

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]

❌ Pipeline failed at stage build after 12m 51s.

No log available