feat(0.2): AI risk subdivision + ctx audit on AI detectors#146
Open
feat(0.2): AI risk subdivision + ctx audit on AI detectors#146
Conversation
Two Track 5 deliverables for the parity-gated 0.2.0 plan. Together
they raise the trust posture of the AI Risk Review surface — both
in how it's classified (Track 5.1) and how it behaves under
cancellation (Track 5.3).
Track 5.1 — AI risk subdivision (Inventory / Hygiene / Regression)
Adds `internal/signals/ai_subdomain.go` with three trust tiers
and a classification for every CategoryAI signal in the manifest:
- Inventory (Tier 1, publicly claimable): direct facts
derived from declared AI surfaces — uncoveredAISurface,
aiPromptVersioning, aiSafetyEvalMissing, capabilityValidationGap,
phantomEvalScenario, aiPolicyViolation, untestedPromptFlow.
- Hygiene (Tier 2, visible but not gating-critical):
heuristic structural patterns — aiPromptInjectionRisk,
aiHardcodedAPIKey, aiToolWithoutSandbox, aiModelDeprecationRisk,
aiFewShotContamination, contextOverflowRisk.
- Regression (Tier 2, eval-data-dependent): fires only when
eval-framework artifacts present — every cost / latency /
hallucination / retrieval / tool-routing / RAG-grounding
signal across the airun catalog.
Public helpers `AISubdomainOf`, `AISubdomainLabel`, and
`AISubdomainTrustBadge` give renderers a single source of truth
for tier vocabulary so PR comment, terminal report, and JSON all
speak the same language. Drift gate test
`TestAISubdomain_AllAISignalsClassified` fails CI if a new AI
signal is added without a tier — closes the "silent dump into
legacy umbrella stanza" failure mode.
Companion doc `docs/product/ai-risk-tiers.md` documents the three
tiers, the public-claim posture per tier, the gating contract
(Tier 1 may be critical; Tier 2 caps at high), and the
add-a-signal recipe.
Track 5.3 — ctx audit on AI detector file walk
Adds `aidetect.DetectContext(ctx, root)` that respects ctx in
the source-walk inner loop — checks `ctx.Err()` every 64 entries,
aborts cleanly when cancelled. The pre-Track-5.3 shape silently
ignored ctx, so a `terrain analyze --timeout 5s` run against a
large repo with AI patterns would still wait for the AI walk to
finish after ctx had been cancelled.
Pipeline call site (`internal/engine/pipeline.go:413`) now uses
DetectContext so cancellation propagates end-to-end. Existing
`aidetect.Detect(root)` is preserved as a backwards-compatible
wrapper that delegates to DetectContext(context.Background()).
New `cancellation_test.go` proves the contract:
- already-cancelled ctx returns within 250ms on a 200-file
fixture (vs ~50ms+ without short-circuit)
- mid-walk cancel after 20ms aborts within 1s on a 1000-file
fixture (vs ~200ms+ without honoring)
- Detect / DetectContext produce identical results on the
same fixture (backwards-compat invariant)
Verification: 48 internal packages pass; cmd tests pass;
make docs-verify clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Terrain AI Risk Review
Decision: PASS — AI surfaces are covered. |
[RISK] Terrain — Merge with caution
Coverage gaps in changed code
4 pre-existing issues on changed files
Recommended tests28 test(s) with exact coverage of 12 impacted unit(s). 2 impacted unit(s) have no covering tests in the selected set.
AI Risk Review
1 advisory finding
Owners: PMCLSF Limitations
Generated by Terrain · Targeted Test ResultsTerrain selected 28 test(s) instead of the full suite.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two Track 5 deliverables for the parity-gated 0.2.0 plan. They raise
the trust posture of the AI Risk Review surface in how findings are
classified (Track 5.1) and how the detector path behaves under
cancellation (Track 5.3).
(Inventory / Hygiene / Regression) classified for every
CategoryAI signal, with a drift gate that fails CI on a missing
classification.
aidetect.DetectContext(ctx, root)honours ctx in thesource-walk inner loop; pipeline calls it; cancellation tests
prove the contract.
What changed
New code:
internal/signals/ai_subdomain.go—AISubdomainenum,aiSubdomainBySignalmap covering 26 AI signals,AISubdomainOf/AISubdomainLabel/AISubdomainTrustBadgehelpers
internal/aidetect/detect.go— newDetectContext(ctx, root),ctx-aware inner walk via
detectFromSourceCtx. ExistingDetect(root)preserved as backwards-compat wrapper.Wiring:
internal/engine/pipeline.go:413— switched toDetectContextso cancellation propagates from
RunPipelineContextend-to-endNew tests:
internal/signals/ai_subdomain_test.go— drift gate +per-tier sample classification + label/badge contract (5 tests)
internal/aidetect/cancellation_test.go— already-cancelledshort-circuit + mid-walk cancel + backwards-compat invariant
(3 tests with realistic fixture sizes)
New doc:
docs/product/ai-risk-tiers.md— the three-tier framing,per-tier signal lists, public-claim posture, gating contract
(Tier 1 may be critical; Tier 2 caps at high), add-a-signal
recipe pointing at the drift gate
Why these together
Both deliverables touch the AI risk surface and both raise the
honest trust posture of
terrain ai run/terrain analyzeinmixed-AI repos. Bundling minimizes review thrash on tightly
related work.
Test plan
go build ./...cleango test ./internal/...— 48 packages pass (incl. 8 new tests)go test ./cmd/...cleanmake docs-verifypassesterrain analyzeon a repo with mixedAI surface types → JSON output carries
aiSubdomainpersignal once renderer wiring lands (separate PR — this PR
establishes the classification + helpers; renderer changes
come in a follow-on)
Plan tracker
Closes Track 5.1 + 5.3 from the parity-gated 0.2.0 plan. Track 5
remaining: 5.6 (per-component timing in --verbose) which is
folded into Track 8.3.
🤖 Generated with Claude Code