Skip to content

Add golden file tests for CLI output #166

@tiulpin

Description

@tiulpin

Area

Other / Not sure

Problem or use case

There are no tests that verify the exact output format of CLI commands. When a contributor changes output.StatusIcon, output.FormatDuration, PrintTable column widths, or any other formatting logic, nothing catches the visual regression.

The existing acceptance tests (.txtar) verify behavior (exit codes, that output contains certain strings) but not exact output formatting. Unit tests cover logic but not rendered output.

This means:

  • A refactor to Printer methods could silently change what users see
  • PR reviewers can't see "before vs after" output diffs
  • Scripts depending on exact output format (e.g., parsing teamcity run list --plain) can break without notice

Proposed solution

Add golden file tests for key output-producing commands. The pattern:

  1. Add a testdata/ directory with .golden files containing expected output
  2. Tests run a command against a mock server and compare stdout to the golden file
  3. An -update flag regenerates golden files when output intentionally changes
var update = flag.Bool("update", false, "update golden files")

func TestAgentList_output(t *testing.T) {
    ts := cmdtest.SetupMockClient(t)
    var buf bytes.Buffer
    ts.Factory.Printer.Out = &buf

    // run command...

    golden := filepath.Join("testdata", t.Name()+".golden")
    if *update {
        os.WriteFile(golden, buf.Bytes(), 0o644)
        return
    }
    expected, _ := os.ReadFile(golden)
    assert.Equal(t, string(expected), buf.String())
}

Priority commands to cover:

  • agent list / agent view — table + detail formatting
  • run list / run list --plain — table + plain modes
  • run view — status icons, progress, metadata
  • job list / job view — table + detail formatting
  • pool list / pool view — table with nested agents
  • queue list — queued build formatting
  • Error output — UserError with hints

Changes to output will show up as diffs in PRs, making review easier and regressions visible.

Alternatives considered

  • Snapshot testing libraries (e.g., bradleyjkemp/cupaloy) — adds a dependency for something achievable with ~20 lines of helper code
  • Checking for substrings in acceptance tests — already done, but too coarse to catch formatting regressions
  • No action — acceptable if output stability isn't a goal, but it matters for scripting (--plain, --json) and contributor confidence

Contribution

  • I'd be willing to submit a PR for this feature

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions