feat(0.2): detector capability metadata + missing-input diagnostics (Tracks 9.1/9.3)#155
Open
feat(0.2): detector capability metadata + missing-input diagnostics (Tracks 9.1/9.3)#155
Conversation
…Tracks 9.1/9.3)
Two paired Track 9 deliverables that lift detector self-description
from "what does it emit" to "what does it consume" — and surface
input gaps as visible diagnostics instead of silent zero-output.
Track 9.1 — Capability metadata extension
Adds five new fields to DetectorMeta:
RequiresRuntime — needs RuntimeStats from runtime artifact
ingestion (--runtime junit.xml / jest.json)
RequiresBaseline — needs Baseline snapshot pointer (--baseline)
RequiresEvalArtifact — needs EvalRuns from Promptfoo / DeepEval
/ Ragas adapter ingestion
ContextAware — honors ctx.Err() in inner loops (descriptive
today; surfaced in `terrain doctor`
cancellation posture)
Experimental — detector implementation not yet stable
(distinct from manifest-level signal status)
All zero-default; existing detectors continue working unchanged.
New detectors that genuinely consume these inputs declare them
via the metadata so the missing-input check knows what to flag.
Track 9.3 — Missing-input diagnostics
New `safeDetectChecked(reg, snap, fn)` — the registry's canonical
detector-invocation path. Pre-Track-9.3 a runtime-needing detector
on a no-runtime snapshot would silently emit zero signals;
adopters had no way to know whether the detector ran-and-found-
nothing or ran-but-was-blind.
When `missingInputs(meta, snap)` returns non-empty, the helper
returns a single SignalDetectorMissingInput marker per affected
detector, with the explanation listing every flag the user needs
to add (Oxford-comma joined). The actual detector body is
skipped — no panic, no waste.
All call sites in detector_registry.go (Run + RunWithGraph
Phase 1/2/3) routed through safeDetectChecked. The check
composes with safeDetect's panic recovery: a panicking detector
with sufficient inputs still produces detectorPanic; a panicking
detector with missing inputs is shielded by the early-return.
New SignalDetectorMissingInput type registered in the manifest
+ signal catalog so ValidateSnapshot accepts the marker (same
posture as detectorPanic).
Coverage
7 new tests (missing_input_test.go):
- happy path (detector runs when no inputs required)
- missing runtime → diagnostic with --runtime flag named
- runtime present (RuntimeStats on TestFile) → detector runs
- missing baseline → diagnostic with --baseline flag named
- missing eval artifact → diagnostic with promptfoo-results
flag named
- multiple missing → one diagnostic listing all three (Oxford)
- joinInputNames covers 0/1/2/3+/4 cases including the Oxford
comma fix
Verification: full Go test suite green; make docs-verify clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
[RISK] Terrain — Merge with caution
Coverage gaps in changed code
8 pre-existing issues on changed files
Recommended tests8 test(s) with exact coverage of 21 impacted unit(s). 11 impacted unit(s) have no covering tests in the selected set.
AI Risk Review
2 advisory findings
Owners: PMCLSF Limitations
Generated by Terrain · Targeted Test ResultsTerrain selected 8 test(s) instead of the full suite.
|
Terrain AI Risk Review
Decision: PASS — AI surfaces are covered. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two paired Track 9 deliverables that lift detector self-description
from "what does it emit" to "what does it consume" — and surface
input gaps as visible diagnostics instead of silent zero-output.
fields: RequiresRuntime, RequiresBaseline, RequiresEvalArtifact,
ContextAware, Experimental. All zero-default; existing detectors
unchanged.
safeDetectCheckedemits a single SignalDetectorMissingInput marker per affected
detector when its declared inputs aren't satisfied. Skips
invocation; no panic; explanation names the exact flags to add.
What changed
New code:
DetectorMeta.{RequiresRuntime,RequiresBaseline,RequiresEvalArtifact,ContextAware,Experimental}— 5 new fields, all zero-default
safeDetectChecked(reg, snap, fn)— canonical invocation path,composes input check with existing safeDetect panic recovery
missingInputs(meta, snap)— returns user-facing flag namesfor each missing input
joinInputNames— Oxford-comma string join (with testscovering 0/1/2/3+ cases)
New signal type:
SignalDetectorMissingInputregistered in signal_types.go,signal_catalog.go, manifest.go
Wiring:
safeDetect(reg, ...)call sites in detector_registry.go(Run + RunWithGraph Phase 1/2/3 paths) routed through
safeDetectChecked(reg, snap, ...)New doc:
docs/rules/engine/detector-missing-input.md(regenerated)Test plan
missing_input_test.go— happy path,per-input missing diagnostics, multi-missing aggregation,
join formatting
go test ./...— full suite greenmake docs-verify— generated docs currentPlan tracker
Closes Track 9.1 + 9.3. Track 9 remaining: 9.2 (panic recovery
completion), 9.5 (pipeline architectural separation), 9.6
(registry refactor), 9.7 (truth-verify). All explicitly
post-0.2.0-blocking.
🤖 Generated with Claude Code