Pipeline Config Drift Detector with Automatic Rollback Alerts

[github-actions]

Strategic Improvement

Success rate at 75% (target >85%) with no automated detection when config/template changes degrade performance. Build a system that compares recent pipeline outcomes against historical baselines, detects statistically significant degradation, and alerts with rollback recommendations. Catches bad changes before they affect many pipelines.

Acceptance Criteria

• Maintains rolling baseline of success rate, duration, cost per template/config hash
• Detects degradation using statistical significance test (e.g., >10% drop, p<0.05)
• Emits alert event with: metric name, old vs new value, suspected config change, rollback command
• Integrates with dashboard to show drift trends
• Tested with simulated config changes and synthetic degradation

Context

• Priority: P0
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	full
Max Iterations	30
Triage	Strategic reliability improvement detecting config-driven pipeline degradation; multi-subsystem integration (daemon, intelligence, eventbus, dashboard) with statistical analysis and comprehensive testing required.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24294839494

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (956 lines)

I'll create a comprehensive implementation plan for the Pipeline Config Drift Detector with Automatic Rollback Alerts feature.

Implementation Plan: Pipeline Config Drift Detector

Strategic Context

Current state: Success rate at 75% (target >85%). Configuration changes sometimes degrade performance, but there's no automated detection or warning system before the changes affect many pipelines.

Goal: Build statistical drift detection that catches bad config changes early, preventing widespread impact.

---

Alternatives Considered

Approach 1: Event-driven Baseline System ✓ CHOSEN

• Collect metrics on every pipeline completion via event system
• Store rolling window (e.g., last 100 runs per config) in SQLite
• Calculate baseline automatically from recent successful runs
• Trigger statistical comparison when new runs arrive
• Emit alert event if p<0.05 AND >10% degradation
• Dashboard subscribes to drift alerts, shows trend visualization

Pros:

• Minimal blast radius (new scripts only, hooks existing event system)
• Fits Shipwright's event-driven architecture perfectly
• Easily testable with synthetic event data
• Non-blocking (async detection via daemon/eventbus)
• Reversible (drop tables, remove hooks)

Cons:

• Requires careful tuning of rolling window size
• Cold start problem for new configs (mitigated by 30-sample threshold)

Complexity/Performance: ~400 LOC for detector, <100ms per comparison (SQLite + t-test), minimal overhead

---

Approach 2: Continuous Baseline Monitoring (Rejected)

• Maintain exponential moving average baseline per config
• Trigger alert on cumulative degradation (e.g., 3 consecutive failures)

Rejected because:

• Doesn't meet acceptance criteria (no statistical significance test)
• Higher false positive rate (react to normal variance)
• Baseline perpetually drifts upward/downward
• Harder to reason about and debug

---

Approach 3: ML-based Anomaly Detection (Rejected)

✅ Task Checklist

• Task 1: Create SQLite schema migration file (templates/sql/drift-schema.sql)
• Task 2: Add metric collection hook to pipeline completion (.claude/hooks/pipeline-completion.sh)
• Task 3: Implement statistical comparison function (Welch's t-test with p-value)
• Task 4: Create scripts/sw-drift-detector.sh main detector script with event emission
• Task 5: Integrate detector into daemon subprocess (modify scripts/sw-daemon.sh)
• Task 6: Emit pipeline completion event with config hash and metrics (modify scripts/sw-loop.sh)
• Task 7: Add dashboard API route for drift trends (modify dashboard/server.ts)
• Task 8: Add drift alert event listener to dashboard broadcast
• Task 9: Implement rollback suggestion command (shipwright config rollback <hash>)
• Task 10: Create test suite with synthetic metrics (scripts/sw-drift-detector-test.sh)
• Task 11: Add integration test for end-to-end flow (event → metric → baseline → alert → dashboard)
• Task 12: Update config schema to include drift thresholds (config/drift-config.json)
• Task 13: Add documentation for drift detector configuration and alerts
• Task 14: Run manual validation: simulate config change and verify alert emission
• Task 15: Update CHANGELOG.md and register test in package.json

---

Generated by shipwright pipeline at 2026-04-12T00:30:01Z

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (935 lines)

Architecture Decision Record: Pipeline Config Drift Detector with Automatic Rollback Alerts

Context

Problem: Configuration changes sometimes degrade pipeline performance, but there's no automated detection or early warning system. Current success rate is 75% (target >85%). Bad configs can propagate widely before being caught.

Constraints:

• Shipwright uses an event-driven architecture with an existing event bus (sw-eventbus.sh)
• SQLite persistence layer already exists (sw-db.sh)
• Dashboard server has real-time event subscription capability (dashboard/server.ts)
• Must be reversible and non-blocking (cannot slow down pipeline execution)
• Must fit within the daemon's subprocess model for continuous monitoring
• TypeScript codebase with Node.js / npm / vitest conventions

Success Criteria:

• Detect >15% performance degradation with statistical significance (p<0.05)
• <2% false positive rate on normal variance
• Alert emission within 5 seconds of detection
• Operator can trigger rollback from dashboard

---

Decision

Chosen Approach: Event-driven statistical baseline system with rolling window persistence.

Why This Approach

This design leverages Shipwright's existing architectural strengths:

1. Event system (existing sw-eventbus.sh) handles async metric delivery without blocking pipeline execution
2. SQLite (existing sw-db.sh) provides durable, queryable storage with transactional guarantees for idempotency
3. Daemon subprocess model allows continuous background detection independent of individual pipeline runs
4. Dashboard subscription pattern fits naturally with real-time alerting (no polling needed)

Core Design Principles

Principle	Implementation
Separation of Concerns	Metric collection (hook) → baseline calculation (detector) → alerting (event) → display (dashboard)
Non-blocking Metrics	Async event emission; detector runs independently on schedule, never blocks pipeline
Idempotency	Run IDs prevent duplicate metrics; alert IDs prevent alert storms
Observability	All decisions logged; operators can review raw metrics and understand why alert fired
Reversibility	Can be disabled/uninstalled without schema changes; tables persist for debugging

---

Alternatives Considered

Alternative 1: Continuous Baseline Monitoring (Rejected)

Approach: Maintain exponential moving average baseline per config; trigger alert on cumulative degradation (e.g., 3 consecutive failures).

Pros:

• Lower latency to detect degradation (immediate vs. waiting for statistical sample)
• No cold start problem (baseline available from first run)

Cons:

• Does not meet acceptance criteria: No statistical significance test; baseline perpetually drifts upward/downward
• Higher false positive rate: Reacts to normal variance and transient failures
• Harder to tune: "How many consecutive failures = alert?" requires magic numbers

---

Generated by shipwright pipeline design stage at 2026-04-12T00:33:10Z

[github-actions]

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 30 max iterations, 1 agent(s), model: haiku

[github-actions]

[github-actions]