Stage Parallelization Opportunity Analyzer with Auto-Composition

[github-actions]

Strategic Improvement

Average pipeline duration at ~90 min (target <60 min) with no automated detection of parallelization opportunities. Build analyzer that examines stage dependency graphs, detects independent stages, and composes parallel execution plans. Prioritize duration reduction over cost when budget allows.

Acceptance Criteria

• Analyzes stage dependency graph from pipeline template (which stages read outputs of others)
• Identifies stages with no inter-dependencies that can run in parallel
• Generates composed pipeline JSON with parallel stage groups (use tmux panes or worktrees)
• Estimates time savings based on historical stage durations
• Tested on all 8 pipeline templates (fast, standard, full, hotfix, autonomous, enterprise, cost-aware, deployed)
• Integrates with pipeline composer to apply parallelization automatically when safe

Context

• Priority: P3
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P3: Cost Efficiency

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Directly improves Shipwright's core pipeline orchestration by enabling stage parallelization to reduce 90→<60 min duration; clear acceptance criteria across 8 templates.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24294836411

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (1109 lines)

---

Stage Parallelization Opportunity Analyzer — Implementation Plan

Issue: #392 — Stage Parallelization Opportunity Analyzer with Auto-Composition
Goal: Reduce 90-minute pipelines to <60 minutes by automatically detecting and executing independent stages in parallel
Complexity: Standard
Target Completion: 8-10 implementation tasks

---

Executive Summary

This feature enables Shipwright to automatically analyze pipeline stage dependency graphs, identify independent stages that can run in parallel, and compose optimized pipeline configurations that use tmux panes or worktrees for concurrent execution. The analyzer will integrate with the existing pipeline composer to apply parallelization safely, with time savings estimation and validation against all 8+ pipeline templates.

---

Requirements Analysis & Design Decisions

Requirements Clarity: Minimum Viable Change

• Must have: Dependency graph analysis, parallel stage detection, time estimation
• Should have: Integration with composer, safety validation, template testing
• Nice to have: Cost-aware parallelization (prioritize duration over cost)

Design Alternatives Considered

Alternative 1: Static Template-Based Parallelization (CHOSEN)

Approach: Hard-code stage dependency rules in bash arrays, build DAG per template, compute parallel levels
Trade-offs:

• ✅ Simple, deterministic, fast analysis (<100ms)
• ✅ No external dependencies (pure bash/jq)
• ✅ Testable without runtime execution
• ❌ Requires manual rule updates if stages change
• ❌ No cross-stage output detection

Why chosen: Parallelization is fundamentally constrained by hard rules (intake before build, build before test). These rules are explicit in composer_validate_pipeline(). A DAG built from templates is sufficient for the acceptance criteria.

Alternative 2: Runtime Output Detection

Approach: Instrument stages to emit output manifests, build DAG from actual I/O at runtime
Trade-offs:

• ✅ Detects all real dependencies dynamically
• ❌ Requires stage instrumentation (blast radius)
• ❌ Complex, slow analysis (multi-second)
• ❌ Hard to test without running stages

Why rejected: Over-engineered for current problem. Stages already have implicit ordering constraints (spec_generation → plan → design → build). Runtime detection adds complexity without proportional value.

Alternative 3: Machine Learning Dependency Prediction

Approach: Use neural network on historical stage timings to predict dependencies
Why rejected: No training data yet, high variance in stage duration, false positives on sparse data.

✅ Task Checklist

• Task 1: Implement DAG analysis function with hard-coded stage rules
• Task 2: Implement topological sort and parallel level detection
• Task 3: Implement time estimation and validation logic
• Task 4: Create CLI entry point (sw-stage-analyzer.sh)
• Task 5: Extend pipeline composer with parallelization integration
• Task 6: Update CLAUDE.md documentation with commands and examples
• Task 7: Create comprehensive test suite covering all 8 templates
• Task 8: Add test helper functions and validation assertions
• Task 9: Run end-to-end integration tests with real composer
• Task 10: Final code review, performance tuning, skill documentation

---

Generated by shipwright pipeline at 2026-04-12T00:30:54Z

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (192 lines)

Now I have all the context I need. Let me produce the ADR.

---

Design: Stage Parallelization Opportunity Analyzer with Auto-Composition

Context

Shipwright pipelines execute 14 stages sequentially (intake → spec_generation → plan → design → build → test → review → spec_verification → compound_quality → pr → merge → deploy → validate → monitor). Average pipeline duration is ~90 minutes, with the build stage consuming ~94% of wall time (8506s p50). The goal is to reduce pipeline duration to <60 minutes by identifying and executing independent stages in parallel.

Codebase constraints:

1. Ordering constraints are explicit and sparse. composer_validate_pipeline() at scripts/sw-pipeline-composer.sh:327-402 enforces exactly 5 ordering rules via _check_order():
   • intake → build, build → test, test → pr, plan → build, review → pr
2. Templates are static JSON in templates/pipelines/ (9 files: standard, fast, full, hotfix, autonomous, enterprise, cost-aware, deployed, tdd). Each stage has id, enabled, gate, and config.
3. Composer is the integration point. composer_create_pipeline() at lines 80-167 produces composed-pipeline.json via intelligence engine or template fallback. This is where parallelization hints must be injected.
4. All scripts must be Bash 3.2 compatible — no associative arrays, no readarray, no ${var,,}.
5. CLI dispatch is exec-based in scripts/sw at line 552 — new commands need a case entry.

Key insight: Because build dominates at ~30 minutes and is on the critical path with no parallelizable sibling at its level, the realistic savings come from parallelizing the lighter pre-build stages (spec_generation, plan, design can overlap) and post-build stages (review + spec_verification can overlap). Estimated savings: 15-33% depending on template, bounded by the build stage bottleneck.

Decision

Approach: Static DAG analysis with Kahn's algorithm topological sort, integrated as an opt-in composer extension.

Architecture

Three components with strict dependency direction (library → CLI → composer):

┌───────────────────────────────────────────────────────────────┐
│               sw-pipeline-composer.sh (Modified)               │
│  --parallelize flag → calls apply_parallelization()           │
│  Validates result via existing composer_validate_pipeline()    │
└──────────────────────────┬────────────────────────────────────┘
                           │ sources
┌──────────────────────────┴────────────────────────────────────┐
│                sw-stage-analyzer.sh (New CLI)                  │
│  Commands: analyze, validate, apply                           │
│  Loads template → runs analysis → saves JSON to artifacts/    │
│  Emits events: parallelization.analyzed                       │
└──────────────────────────┬────────────────────────────────────┘
                           │ sources
┌──────────────────────────┴────────────────────────────────────┐
│          scripts/lib/stage-parallelization.sh (New Library)    │
│                                                                │
│  analyze_stage_dependencies(pipeline_json)                     │
│    → {stages[], dependencies{}, ordering_constraints[]}        │
│                                                                │
│  detect_parallel_levels(deps_json, enabled_stages)             │
│    → {levels[[]], critical_path[], max_parallelism}             │
│                                                                │
│  estimate_duration_savings(levels_json, metrics_json?)         │
│    → {sequential_s, parallel_s, savings_percent, confidence}   │
│                                                                │
│  validate_parallelization(levels_json, pipeline_json)          │
│    → {valid, safety_score, errors[], warnings[]}               │
│                                                                │
│  apply_parallelization(pipeline_json, levels_json, mode)       │
│    → modified pipeline JSON with parallel_level annotations    │

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-04-12T00:35:07Z_

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]