Automatic Pipeline Template Evolution Engine

[github-actions]

Strategic Improvement

Templates are static and don't learn from success/failure patterns over time. Build a system that analyzes template performance by project type (language, framework, size), identifies high-performing configurations, and generates evolved templates. Success rate should improve automatically as the system learns.

Acceptance Criteria

• Analyzes last 30 days of pipeline outcomes grouped by (template, project_type)
• Calculates success rate, avg duration, cost per template+project combo
• Generates candidate evolved templates by adjusting: iteration counts, stage timeouts, model routing, gate policies
• Validates candidate templates with A/B test (20% traffic) before promoting
• Persists evolved templates to ~/.shipwright/evolved-templates/ with performance metadata
• CLI command to view/adopt evolved templates: shipwright templates evolve --adopt

Context

• Priority: P2
• Complexity: full
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P2: Intelligence & Learning

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	full
Max Iterations	30
Triage	Strategically important learning system that improves Shipwright's core pipeline orchestration through data-driven template evolution; requires building new analysis/generation subsystem with A/B testing, multiple iterations to refine heuristics, and integration with daemon/cost tracking.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24294838035

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (522 lines)

Now I'll create a comprehensive implementation plan. Let me document my analysis:

Brainstorming & Design Analysis

Requirements Clarity:

• Minimum Viable Change: Collect pipeline metrics, analyze by (template, project_type), generate one evolved template candidate, A/B test it, promote if successful
• Implicit Requirements: System must be non-blocking (don't slow pipelines), must handle incomplete data gracefully, must preserve historical templates
• Acceptance Criteria: All stated in issue + success rate improvement by 5%+ for adopted evolved templates vs baselines

Design Alternatives:

Approach	Complexity	Performance	Maintainability	Blast Radius
A: Batch+JSON Simple daily job, JSON files, manual adoption	Low	Fast	High	Minimal
B: Real-time Bandit Stream metrics, Thompson sampling A/B	High	Slightly slower	Moderate	Medium
C: Hybrid (CHOSEN) Batch analysis + async metrics + decision gates	Medium	Fast	High	Low

Why Hybrid (Option C):

• Collects metrics passively (no pipeline slowdown)
• Batch analysis allows complex reasoning without impacting runtime
• Explicit A/B test gates prevent bad templates reaching production
• Extends existing infrastructure (daemon, config, templates)
• Can evolve to real-time later

---

Automatic Pipeline Template Evolution Engine — Implementation Plan

1. Files to Modify/Create

scripts/sw-template-evolution.sh           (NEW) Main evolution engine + CLI
scripts/sw-template-analyzer.sh            (NEW) 30-day analysis and aggregation
scripts/sw-template-metrics.sh             (NEW) Metric collection from pipeline runs
scripts/sw-template-ab-test.sh             (NEW) A/B test executor and promotion logic
scripts/lib/template-evolution-lib.sh      (NEW) Shared functions
scripts/sw                                 (MODIFY) Add routing for templates subcommands
scripts/sw-templates.sh                    (MODIFY) Add `evolve` subcommand entry point
.claude/schema/evolved-templates.schema.json (NEW) Schema for evolved template metadata
scripts/sw-template-evolution-test.sh      (NEW) Test suite

Config & Storage:
~/.shipwright/evolved-templates/           (NEW) Directory structure:
  ├── metrics/                             Raw and aggregated metrics (30-day rolling)
  ├── analysis/                            Analysis results and performance reports
  ├── candidates/                          Candidate evolved templates under A/B test
  ├── promoted/                            Promoted evolved templates
  └── history.jsonl                        Audit log of all evolutions

✅ Task Checklist

• Task 1: Design Data Schema — Define metrics collection format, aggregation schema, evolved template metadata. (Blocks: 2, 3, 4)
• Task 2: Implement Metrics Collection — Create sw-template-metrics.sh to passively record pipeline outcomes. (Depends: 1)
• Task 3: Implement Analysis Engine — Create sw-template-analyzer.sh to compute success rates, duration, cost per (template, project_type). (Depends: 1, 2)
• Task 4: Implement Evolution Generator — Create sw-template-evolution-engine.sh to synthesize evolved templates from analysis. (Depends: 1, 3)
• Task 5: Implement A/B Testing — Create sw-template-ab-test.sh with statistical decision gates. (Depends: 1, 4)
• Task 6: Integrate Metrics Collection — Add hooks to pipeline.sh to call metrics collection. (Depends: 2)
• Task 7: Integrate CLI — Add shipwright templates evolve command with subcommands (--list, --analyze, --generate, --test, --promote). (Depends: 3, 4, 5)
• Task 8: Implement Storage Layer — Create directory structure and atomic write helpers. (Depends: 1)
• Task 9: Write Tests — Create comprehensive test suite covering happy path and failure modes. (Depends: 2–8)
• Task 10: Documentation & Integration — Update docs, add to MEMORY.md, wire into daemon patrol. (Depends: 9)
• Metrics collected: ~/.shipwright/evolved-templates/metrics/ contains >= 100 runs after 1 day
• Analysis runs: shipwright templates evolve --analyze produces report with success rates, durations
• Evolution generates: shipwright templates evolve --generate standard produces candidate with rationale
• A/B test runs: Candidate receives 20% of matching template traffic for >= 100 runs
• Promotion works: shipwright templates evolve --promote CANDIDATE_ID copies to ~/.shipwright/templates/
• CLI commands: All 7 subcommands work (--list, --analyze, --generate, --test, --promote, --status, --history)
• Tests pass: 40+ test cases, covering happy path + 3+ edge cases
• Docs updated: README section on template evolution, CLI help text
• No regression: All existing pipeline tests pass, no slowdown

---

Generated by shipwright pipeline at 2026-04-12T00:29:20Z

[github-actions]

[github-actions]

❌ Pipeline failed at stage design after 57s.

No log available

[github-actions]

[github-actions]

❌ Pipeline Failed (exit code: 1)

Failure Analysis

Field	Value
Failed Stage	design
Error Type	test_failure
Completed	intake, spec_generation, plan

Stage Timeline

Stage	Status	Duration
intake	✅	1m 45s
spec_generation	✅	1m 0s
plan	✅	2m 49s
design	❌	57s

Last 20 lines of log

▸ Plan posted to issue #391
▸ Claude Code tasks: /home/runner/work/shipwright/shipwright/.claude/tasks.md
▸ Validating plan (attempt 1/2)...
✓ Plan validation passed
https://github.com/sethdford/shipwright/issues/391#issuecomment-4230441792
⚠ No check run ID found for stage 'plan'
▸ Persisting 3 artifact(s) after stage plan...
evt-1775953819-341218512
✓ Stage plan complete (2m 49s)
✓ Broadcast discovery: plan (*.md)

▸ Stage: design [4/11]
⚠ No check run ID found for stage 'design'
▸ Generating Architecture Decision Record...
✗ Design too short (1 lines) — likely an error, not a real design
https://github.com/sethdford/shipwright/issues/391#issuecomment-4230443050
https://github.com/sethdford/shipwright/issues/391#issuecomment-4230443061
⚠ No check run ID found for stage 'design'
evt-1775953879-2415245040
✗ Pipeline failed at stage: design

View full run

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	full
Max Iterations	30
Triage	Strategic enhancement to core template system enabling data-driven evolution. Improves cost efficiency and reliability through metrics collection, template generation, A/B testing infrastructure, and persistence layer across multiple subsystems.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24295272887

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]