feat(testing): Add Golden Response System for Snapshot Testing#1607
Open
Mustafa11300 wants to merge 3 commits intomofa-org:mainfrom
Open
feat(testing): Add Golden Response System for Snapshot Testing#1607Mustafa11300 wants to merge 3 commits intomofa-org:mainfrom
Mustafa11300 wants to merge 3 commits intomofa-org:mainfrom
Conversation
Implements parameterized scenario expansion for the mofa-testing crate.
One scenario template can now expand into many concrete test cases by
substituting {{variable}} placeholders with values from parameter sets.
New components:
- ParameterSet: named variable bindings for one test variant
- ParameterMatrix: Cartesian product expansion with safety limits
- ParameterizedScenario: template + parameter sets -> expanded scenarios
- ParameterizedScenarioFile: YAML/TOML/JSON file-backed loading
Includes:
- 30+ comprehensive tests covering expansion, substitution, file
loading, execution, edge cases, and error handling
- Example scenarios in examples/parameterized_test/ with YAML, TOML,
and matrix expansion demonstrations
- README with usage guide and code samples
Closes #<ISSUE_NUMBER>
Implements golden response (snapshot) testing for the mofa-testing crate. Agent outputs are recorded as baselines, then future runs are compared against them to detect regressions automatically. New components: - GoldenSnapshot: serializable record of turn outputs (JSON/YAML) - GoldenStore: filesystem-backed snapshot persistence - GoldenTestConfig: strict (validate) vs. update (record) modes - GoldenDiff: structured per-field diff reporting - Normalizer trait + WhitespaceNormalizer, RegexNormalizer, NormalizerChain - run_golden_test: end-to-end golden test runner integrated with TestReport - compare_golden: standalone comparison engine Includes: - 30+ comprehensive tests covering serialization, store operations, diff detection, normalizers, update/strict mode, multi-turn, and tool call verification - Example golden snapshots and README in examples/golden_response_test/ - tempfile added as dev-dependency for test isolation Closes #<ISSUE_NUMBER>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔍 Description
Adds golden response (snapshot) testing to the
mofa-testingcrate. Agent outputs are recorded as baselines, then future runs are compared against them to detect regressions automatically.This is Issue 4 from the testing platform roadmap (Test Definition Layer, Phase 2).
Closes #1604
Depends on #1599
📌 Changes
New:
tests/src/golden.rsCore golden response module with the following public components:
GoldenSnapshotGoldenTurnSnapshotGoldenStore.golden.jsonfilesGoldenTestConfigGoldenCompareModeStrictorUpdateGoldenCompareResultGoldenDiffGoldenErrorNormalizertraitWhitespaceNormalizerRegexNormalizerNormalizerChaindefault_chain()handles UUIDs + timestamps + whitespacecompare_golden()run_golden_test()TestReportComparison fields:
Modes:
Modified:
tests/src/lib.rspub mod goldenmodule registrationModified:
tests/Cargo.tomltempfile = { workspace = true }as dev-dependency for test isolationNew:
tests/tests/golden_tests.rs30+ comprehensive tests covering:
New:
examples/golden_response_test/README.mdgoldens/weather_agent.golden.jsongoldens/support_agent.golden.json🧪 Testing
All new functionality is covered by
tests/tests/golden_tests.rswith 30+ test cases.Key test categories:
💡 Usage Example
Record a golden baseline
Validate against golden
With normalizers (ignore UUIDs/timestamps)
CI Workflow
✅ Checklist
lib.rswith public re-exportsgolden_tests.rs)examples/golden_response_test/)GoldenErrortempfiledev-dependency uses workspace version