Skip to content

Add snapshot/golden test for TerminalReporter output stability #63

@Zious11

Description

@Zious11

Background

Surfaced by PR review of #5 — pr-test-analyzer flagged that default_rendering_unchanged_when_mitre_flag_off (in tests/reporter_tests.rs) checks substrings (`MITRE: T1046`, absence of em-dash, absence of `## Uncategorized`) but does not assert byte-identical output against a known-good baseline.

What snapshot testing would buy

  • Catches incidental whitespace/format drift in the FINDINGS block, ANALYZER section, etc.
  • Forces explicit acknowledgment of any intentional render change (you regenerate the snapshot).
  • Covers behaviors the substring tests can't easily express (e.g., the section-header underline length, color-code byte sequences when `use_color = true`).

What snapshot testing costs

  • Friction on every legitimate format change (must regenerate the snapshot).
  • A new dev-dependency (most likely `insta` — de-facto Rust crate for snapshot testing).
  • Snapshots committed to git can grow large for analyzers with many summary lines.

Why deferred

Validated against external best-practice (Perplexity inconclusive on `insta` adoption rate); the trade-off is real — current substring tests cover the contract that matters today, and snapshot testing is the right tool for "this output should never change without me noticing," which is a different concern from "this output is correct."

Suggested approach if undertaken

  1. Add `insta = "1"` to `[dev-dependencies]`.
  2. Cover the default flat rendering against a fixture finding set first — this is the highest-stability path.
  3. Cover the grouped rendering second.
  4. Document in `tests/reporter_tests.rs` that intentional format changes require `cargo insta review` to accept the new snapshot.

Acceptance criteria

  • Snapshot tests for default + grouped FINDINGS blocks.
  • README or CONTRIBUTING note about regenerating snapshots.
  • No regression in existing substring tests (they remain as semantic assertions).

Metadata

Metadata

Assignees

No one assigned

    Labels

    reporterOutput/export formatstestTest coverage

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions