Skip to content

refactor: redesign test guardrail from line-ratio to co-modification check #40

@jdelfino

Description

@jdelfino

Problem

The current guardrail/test-ratio check counts lines of test code vs lines of implementation code and compares against a threshold (default 0.5). This is a poor signal:

  • Test boilerplate/setup inflates line counts without meaningful coverage
  • A one-line behavior change may legitimately need many test lines (or none)
  • The ratio says nothing about whether changed code actually has updated tests
  • Additionally, the current implementation has a bug where test lines are never counted (isCodeFile excludes test files, so the isTestFile branch is dead code)

New approach: co-modification check

Replace line-ratio counting with a test co-modification check: when an implementation file is changed in a PR, verify that a corresponding test file was also changed in the same PR.

This answers "did you update tests for what you changed?" — a more actionable and meaningful signal than a line ratio.

Pairing heuristic

Match by base filename:

  • lib/foo.js pairs with any foo.test.js / foo.spec.js / test_foo.py found in the PR's changed files
  • Strip test directory prefixes (tests/, __tests__/) and test suffixes (.test., .spec., _test.) to extract the base name
  • A match means the impl file has a co-modified test

Reporting

  • Annotate each unpaired impl file (changed code with no corresponding test change)
  • Summary shows count of paired vs unpaired files
  • action_required if any impl files lack a paired test change (configurable conclusion)
  • success if all impl files have paired tests, or if PR has no impl changes

Configuration changes

  • Remove threshold (no longer applicable)
  • Add optional exclude-paths list for files that legitimately don't need tests (configs, types, generated code)

Edge cases

  • PR with only test files → auto-pass
  • PR with only non-code files (markdown, yaml, etc.) → auto-pass
  • Files in exclude-paths → skip (don't count as unpaired)
  • PR approval remains a universal override (existing behavior)

Files to modify

  • .github/agent-workflow/scripts/guardrail-test-ratio.js — rewrite core logic
  • .github/agent-workflow/scripts/lib/file-patterns.js — add testBaseName() helper to extract base name from test/impl files for pairing
  • .github/agent-workflow/config.yaml — remove threshold, add exclude-paths
  • tests/lib/file-patterns.test.js — add tests for new helper
  • New: tests/guardrail-test-ratio.test.js — unit test the guardrail logic directly

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions