-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Area
Other / Not sure
Problem or use case
There are no tests that verify the exact output format of CLI commands. When a contributor changes output.StatusIcon, output.FormatDuration, PrintTable column widths, or any other formatting logic, nothing catches the visual regression.
The existing acceptance tests (.txtar) verify behavior (exit codes, that output contains certain strings) but not exact output formatting. Unit tests cover logic but not rendered output.
This means:
- A refactor to
Printermethods could silently change what users see - PR reviewers can't see "before vs after" output diffs
- Scripts depending on exact output format (e.g., parsing
teamcity run list --plain) can break without notice
Proposed solution
Add golden file tests for key output-producing commands. The pattern:
- Add a
testdata/directory with.goldenfiles containing expected output - Tests run a command against a mock server and compare stdout to the golden file
- An
-updateflag regenerates golden files when output intentionally changes
var update = flag.Bool("update", false, "update golden files")
func TestAgentList_output(t *testing.T) {
ts := cmdtest.SetupMockClient(t)
var buf bytes.Buffer
ts.Factory.Printer.Out = &buf
// run command...
golden := filepath.Join("testdata", t.Name()+".golden")
if *update {
os.WriteFile(golden, buf.Bytes(), 0o644)
return
}
expected, _ := os.ReadFile(golden)
assert.Equal(t, string(expected), buf.String())
}Priority commands to cover:
agent list/agent view— table + detail formattingrun list/run list --plain— table + plain modesrun view— status icons, progress, metadatajob list/job view— table + detail formattingpool list/pool view— table with nested agentsqueue list— queued build formatting- Error output —
UserErrorwith hints
Changes to output will show up as diffs in PRs, making review easier and regressions visible.
Alternatives considered
- Snapshot testing libraries (e.g.,
bradleyjkemp/cupaloy) — adds a dependency for something achievable with ~20 lines of helper code - Checking for substrings in acceptance tests — already done, but too coarse to catch formatting regressions
- No action — acceptable if output stability isn't a goal, but it matters for scripting (
--plain,--json) and contributor confidence
Contribution
- I'd be willing to submit a PR for this feature
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request