Manual QA: test the pred CLI tool

[GiggleLiu]

Objective

Manually test the pred CLI tool end-to-end. This covers all subcommands, error handling, piping, and JSON output modes.

Setup

# Build the CLI in release mode
make cli

# Verify installation
./target/release/pred --version
./target/release/pred --help

Or use cargo run with dev profile:

alias pred='cargo run --manifest-path problemreductions-cli/Cargo.toml --'

Testing Checklist

1. Discovery commands

• pred list — lists all registered problems with aliases
• pred list --json — outputs valid JSON to stdout
• pred show MIS — shows variants, size fields, and reductions for MIS
• pred show MIS --json — JSON output
• pred show MIS/UnitDiskGraph — shows a specific graph variant ⚠️ Bug: ignores variant, shows same as pred show MIS
• pred from MIS — shows 1-hop outgoing neighbors
• pred from MIS --hops 2 — shows 2-hop outgoing neighbors
• pred to QUBO — shows 1-hop incoming neighbors
• pred to QUBO --hops 2 — shows 2-hop incoming neighbors

2. Path finding

• pred path MIS QUBO — finds cheapest path (minimize steps)
• pred path MIS QUBO --all — lists all available paths
• pred path MIS QUBO --cost minimize:num_variables — custom cost function
• pred path MIS QUBO -o path.json — saves path for use with pred reduce --via
• pred path MIS MIS — same source and target (edge case)
• pred path SAT QUBO — multi-step reduction path

3. Problem creation

• Graph problems:
  • pred create MIS --graph 0-1,1-2,2-3 -o problem.json
  • pred create MIS --graph 0-1,1-2 --weights 2,1,3
  • pred create MVC --graph 0-1,1-2,2-0
  • pred create MaxCut --graph 0-1,1-2,2-3
• SAT problems:
  • pred create SAT --num-vars 3 --clauses "1,2;-1,3"
  • pred create 3SAT --num-vars 4 --clauses "1,2,3;-1,-2,4"
• QUBO:
  • pred create QUBO --matrix "1,0.5;0.5,2"
• KColoring:
  • pred create KColoring --graph 0-1,1-2,2-0 --k 3
• Factoring:
  • pred create Factoring --target 15 --m 4 --n 4
• Random generation:
  • pred create MIS --random --num-vertices 10 --edge-prob 0.3
  • pred create MIS --random --num-vertices 10 --seed 42 — reproducible (run twice, compare)
• Geometry variants:
  • pred create MIS/KingsSubgraph/i32 --positions "0,0;1,0;1,1;0,1"
  • pred create MIS/UnitDiskGraph/i32 --positions "0.0,0.0;1.0,0.0;0.5,0.8" --radius 1.5

4. Evaluate

• pred evaluate problem.json --config 1,0,1,0 — evaluates a config
• pred create MIS --graph 0-1,1-2 | pred evaluate - --config 1,0,1 — piping from stdin
• Test invalid config length (should error gracefully)

5. Reduce

• pred reduce problem.json --to QUBO -o reduced.json — direct reduction
• pred reduce problem.json --via path.json -o reduced.json — via saved path
• pred create MIS --graph 0-1,1-2 | pred reduce - --to QUBO --json — piping (requires --json)
• pred inspect reduced.json — inspect the reduction bundle

6. Solve

• pred solve problem.json — ILP solver (default)
• pred solve problem.json --solver brute-force — brute-force solver
• pred solve reduced.json — solve a reduction bundle (solution mapped back)
• pred create MIS --graph 0-1,1-2 | pred solve - — piping
• pred solve problem.json --timeout 10 — timeout option
• Compare ILP and brute-force results on the same instance — should agree

7. Full closed-loop workflow

Test the complete pipeline end-to-end:

# Create → Solve → Verify
pred create MIS --graph 0-1,1-2,2-3,3-0 -o problem.json
pred solve problem.json -o result.json
# Check that the solution config is valid:
pred evaluate problem.json --config <solution_from_result>

# Create → Reduce → Solve → Verify
pred create MIS --graph 0-1,1-2,2-3 -o problem.json
pred path MIS QUBO -o path.json
pred reduce problem.json --via path.json -o reduced.json
pred solve reduced.json -o result.json
# Solution should be mapped back to MIS space

# Full pipe chain (requires --json on reduce)
pred create MIS --graph 0-1,1-2,2-3 | pred reduce - --to QUBO --json | pred solve -

8. JSON output and piping

• Every command with --json produces valid JSON (pipe through jq . to verify)
• Every command with -o file.json writes valid JSON to the file
• Stdin reading with - works for: evaluate, reduce, solve, inspect
• --quiet suppresses stderr info messages while still producing stdout output
• ⚠️ Bug: reduce/solve/evaluate don't auto-detect pipe — create outputs JSON when stdout is not a TTY, but other commands always output human-readable text unless --json is specified

9. Error handling

• Unknown problem name → clear error message with suggestions
• Missing required args (e.g., pred create MIS without --graph) → helpful per-problem help
• Invalid edge format → clear error
• Non-existent reduction path → informative message
• Invalid JSON input file → graceful error
• Mismatched config length in pred evaluate → clear error
• ⚠️ Bug: variant ambiguity — pred from MIS/UnitDiskGraph fails because both i32 and One weights match. Should prefer default weight.

10. Shell completions

• ⚠️ Bug: eval "$(pred completions)" fails on macOS zsh with compdef:153: _comps: assignment to invalid subscript range
• Tab completion works for subcommands and problem names

11. Known display issues

• ⚠️ Bug: header in from/to doesn't show variant — pred from MIS/UnitDiskGraph/i32 shows "MaximumIndependentSet" instead of "MaximumIndependentSet/UnitDiskGraph"
• ⚠️ Bug: "N reachable problems" counts unique names, not variant nodes — tree shows 4 nodes but says "3 reachable problems" because MIS variants are collapsed
• ⚠️ Bug: SpinGlass couplings/fields can't be floats — pred create SpinGlass/f64 --couplings 1.0,-0.5 fails with "invalid digit found in string"
• ⚠️ Bug: --hops default listed twice in help: [default: 1] [default: 1]

Reporting

For each failed item, please note:

1. The exact command run
2. The actual output / error
3. The expected behavior

File findings as comments on this issue or as separate bug issues referencing this one.

[GiggleLiu]

@isPANN These test steps requires human input. It would be great if you could help.

[isPANN]

pred show MIS/UnitDiskGraph produces the same result as pred show MIS.

[isPANN]

pred to MIS meaning "the problem that MIS can reduce to" feels a bit ambiguous — it could also be interpreted as "the problem that can be reduced to MIS."

[GiggleLiu]

pred to MIS meaning "the problem that MIS can reduce to" feels a bit ambiguous — it could also be interpreted as "the problem that can be reduced to MIS."

I think "pred to MIS" should return problems that can be reduced to MIS. Do you have any suggestions?

[isPANN]

pred to MIS meaning "the problem that MIS can reduce to" feels a bit ambiguous — it could also be interpreted as "the problem that can be reduced to MIS."

I think "pred to MIS" should return problems that can be reduced to MIS. Do you have any suggestions?

I agree with this.
In addition to fixing the directions of these two commands, we also need to unify the command naming with the output in pred show.
Currently, the output of pred show MIS is:

Reduces to (10):      # "Reduces to" is not a proper label, actually the result of `pred from MIS`
  MaximumIndependentSet {graph=SimpleGraph, weight=i32} → ILP (default)
....

Reduces from (9):   # "Reduces from" is not a proper label, actually the result of `pred to MIS`
  MaximumIndependentSet {graph=KingsSubgraph, weight=i32} ← MaximumIndependentSet {graph=UnitDiskGraph, weight=i32}
....

[isPANN]

Variants (2):
  {graph=SimpleGraph, k=KN}
  {graph=SimpleGraph, k=K3}

Here I think we need to show the name of the variants, like KColoring/KN.

[isPANN]

The help output of pred create contains too much information, making it hard for users to identify which parameters are relevant to their specific problem. Could we separate the accepted parameters for each problem type? Just like this:

pred create MIS --help

using a command like this, only the help information of this particular problem (here is MIS) will be printed.

[GiggleLiu]

QA Test Results (2026-02-28, branch fix/variant-display)

Tested all subcommands end-to-end. Note: the --edges flag in the original checklist has been renamed to --graph. The --bits-m/--bits-n flags are now --m/--n.

Bugs Found

1. Variant ambiguity error (Critical)

Commands fail when a partial variant matches multiple registered variants:

$ pred from MIS/UnitDiskGraph
Error: Ambiguous variant for MaximumIndependentSet with values ["UnitDiskGraph"].
Matches: [{"graph": "UnitDiskGraph", "weight": "i32"}, {"graph": "UnitDiskGraph", "weight": "One"}]

Affects: from, to, create with any geometry variant (MIS/UnitDiskGraph, MIS/KingsSubgraph)
Workaround: Specify full variant MIS/UnitDiskGraph/i32
Fix needed: When ambiguous, prefer the variant closest to defaults (e.g., i32 over One)

2. pred show ignores variant filter

pred show MIS/UnitDiskGraph/i32 shows the exact same output as pred show MIS — all 7 variants and all 15+14 reductions. It should either filter reductions to those relevant to the specified variant, or highlight the specified variant.

3. Header doesn't show variant in from/to

$ pred from MIS/UnitDiskGraph/i32
MaximumIndependentSet — 1-hop neighbors (outgoing)  # should show MaximumIndependentSet/UnitDiskGraph

(Fix already prepared locally — uses fmt_node to include variant in header)

4. "N reachable problems" counts unique names, not nodes

$ pred to MIS
MaximumIndependentSet
├── MaximumIndependentSet/UnitDiskGraph
├── MaximumIndependentSet/One
├── MaximumSetPacking
└── MinimumVertexCover

3 reachable problems in 1 hops   # shows 4 nodes but says 3

MIS/UnitDiskGraph and MIS/One are collapsed into 1 "MaximumIndependentSet". Should either say "4 reachable nodes" or "3 reachable problems (4 variant nodes)".

5. reduce doesn't auto-detect pipe for JSON output

create auto-outputs JSON when stdout is not a TTY, but reduce, solve, and evaluate do not:

$ pred create MIS --graph 0-1,1-2 | pred reduce - --to QUBO | pred solve -
Error: Failed to parse JSON   # reduce outputs human text, not JSON

Workaround: pred create ... | pred reduce - --to QUBO --json | pred solve -

6. SpinGlass doesn't support float couplings/fields

$ pred create SpinGlass --graph 0-1,1-2 --couplings 1.0,-0.5 --fields 0.1,0.2,0.3
Error: invalid digit found in string

Even pred create SpinGlass/f64 ... fails. The parser always uses i32. Should auto-detect or accept float syntax.

7. --hops default listed twice in help

Options:
  --hops <HOPS>  Number of hops to explore [default: 1] [default: 1]

8. Shell completions fail on macOS zsh

$ eval "$(pred completions)"
compdef:153: _comps: assignment to invalid subscript range

Unstaged changes on branch (design concern)

There are work-in-progress changes in src/rules/graph.rs that change the default variant from i32 to One. This completely changes what pred from MIS returns (shows only internal variant casts instead of cross-problem reductions like ILP, QUBO). The default variant question needs resolution before merging.

What's working well

• All --json outputs produce valid JSON
• -o file.json works correctly for all commands
• create supports all problem types with correct flags
• solve ILP and brute-force agree on optimal values
• Random generation with --seed is reproducible
• Error messages for unknown problems, invalid edges, mismatched config lengths are clear
• Per-problem help (pred create MIS without flags) works well
• from/to direction semantics and help text are correct
• Full pipeline works: create → path → reduce → solve → evaluate
• inspect correctly identifies problem types and reduction bundles
• --quiet suppresses stderr info messages

Updated checklist items

The original checklist needs these flag updates:

• --edges → --graph
• --bits-m → --m, --bits-n → --n
• --edge-weights flag is needed for MaxCut/TSP (not --weights)