Skip to content

feat(0.2): detector capability metadata + missing-input diagnostics (Tracks 9.1/9.3)#155

Open
pmclSF wants to merge 1 commit intomainfrom
feat/0.2-track9-capability-metadata
Open

feat(0.2): detector capability metadata + missing-input diagnostics (Tracks 9.1/9.3)#155
pmclSF wants to merge 1 commit intomainfrom
feat/0.2-track9-capability-metadata

Conversation

@pmclSF
Copy link
Copy Markdown
Owner

@pmclSF pmclSF commented May 3, 2026

Summary

Two paired Track 9 deliverables that lift detector self-description
from "what does it emit" to "what does it consume" — and surface
input gaps as visible diagnostics instead of silent zero-output.

  • Track 9.1 — Capability metadata. Adds 5 new DetectorMeta
    fields: RequiresRuntime, RequiresBaseline, RequiresEvalArtifact,
    ContextAware, Experimental. All zero-default; existing detectors
    unchanged.
  • Track 9.3 — Missing-input diagnostics. New safeDetectChecked
    emits a single SignalDetectorMissingInput marker per affected
    detector when its declared inputs aren't satisfied. Skips
    invocation; no panic; explanation names the exact flags to add.

What changed

New code:

  • DetectorMeta.{RequiresRuntime,RequiresBaseline,RequiresEvalArtifact,ContextAware,Experimental}
    — 5 new fields, all zero-default
  • safeDetectChecked(reg, snap, fn) — canonical invocation path,
    composes input check with existing safeDetect panic recovery
  • missingInputs(meta, snap) — returns user-facing flag names
    for each missing input
  • joinInputNames — Oxford-comma string join (with tests
    covering 0/1/2/3+ cases)

New signal type:

  • SignalDetectorMissingInput registered in signal_types.go,
    signal_catalog.go, manifest.go

Wiring:

  • All 6 safeDetect(reg, ...) call sites in detector_registry.go
    (Run + RunWithGraph Phase 1/2/3 paths) routed through
    safeDetectChecked(reg, snap, ...)

New doc: docs/rules/engine/detector-missing-input.md (regenerated)

Test plan

  • 7 new tests in missing_input_test.go — happy path,
    per-input missing diagnostics, multi-missing aggregation,
    join formatting
  • go test ./... — full suite green
  • make docs-verify — generated docs current

Plan tracker

Closes Track 9.1 + 9.3. Track 9 remaining: 9.2 (panic recovery
completion), 9.5 (pipeline architectural separation), 9.6
(registry refactor), 9.7 (truth-verify). All explicitly
post-0.2.0-blocking.

🤖 Generated with Claude Code

…Tracks 9.1/9.3)

Two paired Track 9 deliverables that lift detector self-description
from "what does it emit" to "what does it consume" — and surface
input gaps as visible diagnostics instead of silent zero-output.

Track 9.1 — Capability metadata extension
  Adds five new fields to DetectorMeta:

    RequiresRuntime       — needs RuntimeStats from runtime artifact
                            ingestion (--runtime junit.xml / jest.json)
    RequiresBaseline      — needs Baseline snapshot pointer (--baseline)
    RequiresEvalArtifact  — needs EvalRuns from Promptfoo / DeepEval
                            / Ragas adapter ingestion
    ContextAware          — honors ctx.Err() in inner loops (descriptive
                            today; surfaced in `terrain doctor`
                            cancellation posture)
    Experimental          — detector implementation not yet stable
                            (distinct from manifest-level signal status)

  All zero-default; existing detectors continue working unchanged.
  New detectors that genuinely consume these inputs declare them
  via the metadata so the missing-input check knows what to flag.

Track 9.3 — Missing-input diagnostics
  New `safeDetectChecked(reg, snap, fn)` — the registry's canonical
  detector-invocation path. Pre-Track-9.3 a runtime-needing detector
  on a no-runtime snapshot would silently emit zero signals;
  adopters had no way to know whether the detector ran-and-found-
  nothing or ran-but-was-blind.

  When `missingInputs(meta, snap)` returns non-empty, the helper
  returns a single SignalDetectorMissingInput marker per affected
  detector, with the explanation listing every flag the user needs
  to add (Oxford-comma joined). The actual detector body is
  skipped — no panic, no waste.

  All call sites in detector_registry.go (Run + RunWithGraph
  Phase 1/2/3) routed through safeDetectChecked. The check
  composes with safeDetect's panic recovery: a panicking detector
  with sufficient inputs still produces detectorPanic; a panicking
  detector with missing inputs is shielded by the early-return.

  New SignalDetectorMissingInput type registered in the manifest
  + signal catalog so ValidateSnapshot accepts the marker (same
  posture as detectorPanic).

Coverage

7 new tests (missing_input_test.go):
  - happy path (detector runs when no inputs required)
  - missing runtime → diagnostic with --runtime flag named
  - runtime present (RuntimeStats on TestFile) → detector runs
  - missing baseline → diagnostic with --baseline flag named
  - missing eval artifact → diagnostic with promptfoo-results
    flag named
  - multiple missing → one diagnostic listing all three (Oxford)
  - joinInputNames covers 0/1/2/3+/4 cases including the Oxford
    comma fix

Verification: full Go test suite green; make docs-verify clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 3, 2026

[RISK] Terrain — Merge with caution

High-severity gaps found in changed code.

Metric Value
Changed files 7 (4 source · 1 test)
Impacted units 32
Protection gaps 11
Tests selected 8 of 773 (1% of suite)

Coverage gaps in changed code

  • internal/models/signal_catalog.go [MED] — Exported function KnownSignalTypes has no observed test coverage.
    → Add unit tests for exported function KnownSignalTypes — this is public API surface.
  • internal/models/signal_catalog.go [MED] — Exported function SignalCatalogEntry has no observed test coverage.
    → Add unit tests for exported function SignalCatalogEntry — this is public API surface.
  • internal/models/signal_catalog.go [MED] — Exported function SignalSource has no observed test coverage.
    → Add unit tests for exported function SignalSource — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function All has no observed test coverage.
    → Add unit tests for exported function All — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function ByDomain has no observed test coverage.
    → Add unit tests for exported function ByDomain — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function Detectors has no observed test coverage.
    → Add unit tests for exported function Detectors — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function Len has no observed test coverage.
    → Add unit tests for exported function Len — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function Register has no observed test coverage.
    → Add unit tests for exported function Register — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function Run has no observed test coverage.
    → Add unit tests for exported function Run — this is public API surface.
  • internal/signals/detector_registry.go [MED] — Exported function RunDomain has no observed test coverage.
    → Add unit tests for exported function RunDomain — this is public API surface.
  • ...and 1 more (1 medium)
8 pre-existing issues on changed files
  • docs/signals/manifest.json [MED] — [aiModelDeprecationRisk] model tag gpt-3.5-turbo is a moving alias; pin a dated variant
  • docs/signals/manifest.json [MED] — [aiModelDeprecationRisk] model tag gpt-4 resolves to whatever the provider currently maps it to; pin a dated variant (e.g. gpt-4-0613)
  • internal/signals/manifest.go [MED] — [aiModelDeprecationRisk] model tag gpt-3.5-turbo is a moving alias; pin a dated variant
  • internal/signals/manifest.go [MED] — [aiModelDeprecationRisk] model tag gpt-4 resolves to whatever the provider currently maps it to; pin a dated variant (e.g. gpt-4-0613)
  • internal/models/signal_catalog.go [HIGH] — [blastRadiusHotspot] Changes to this file propagate to 2142 tests (1510 direct, 632 indirect). High blast radius increases regression risk.
  • ...and 3 more

Recommended tests

8 test(s) with exact coverage of 21 impacted unit(s). 11 impacted unit(s) have no covering tests in the selected set.

Test Confidence Why
internal/engine/registry_error_test.go exact exact coverage of DetectorMeta, DetectorRegistration, Domain + 1 more
internal/engine/registry_test.go exact exact coverage of Domain, EvidenceType
internal/measurement/measurement_test.go exact exact coverage of NewRegistry
internal/signals/detector_registry_test.go exact exact coverage of IsKnownSignalType
internal/signals/manifest_rule_docs_test.go exact exact coverage of AllSignalTypes, Manifest, SignalStatus + 8 more
internal/signals/manifest_test.go exact exact coverage of SignalCatalog
internal/signals/missing_input_test.go exact exact coverage of DetectorMeta, DetectorRegistration, Domain
internal/structural/structural_test.go exact exact coverage of EvidenceType

AI Risk Review

Scenarios: 0 of 16 selected

2 advisory findings
  • docs/signals/manifest.json:1087 — Model tag is sunset or floats — the next API call could break or silently re-resolve.
    → Pin to a dated model variant (e.g. gpt-4-0613) or upgrade to a current tier.
  • internal/signals/manifest.go:841 — Model tag is sunset or floats — the next API call could break or silently re-resolve.
    → Pin to a dated model variant (e.g. gpt-4-0613) or upgrade to a current tier.

Owners: PMCLSF

Limitations
  • No coverage artifacts provided; protection gaps reflect missing data, not measured absence. Provide --coverage to improve accuracy.
  • Mixed test cultures reduce cross-framework optimization confidence. Consider standardizing on fewer frameworks.

Generated by Terrain · terrain pr --json for machine-readable output

Targeted Test Results

Terrain selected 8 test(s) instead of the full suite.

  • Go tests: passed

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 3, 2026

Terrain AI Risk Review

Metric Value
AI surfaces 13
Eval scenarios 16
Impacted scenarios 0
Uncovered surfaces 13

Decision: PASS — AI surfaces are covered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant