Skip to content

feat(testing): add agent test DSL and scenario loader#1602

Open
Mustafa11300 wants to merge 1 commit intomofa-org:mainfrom
Mustafa11300:feat/issue2-agent-test-dsl-revive
Open

feat(testing): add agent test DSL and scenario loader#1602
Mustafa11300 wants to merge 1 commit intomofa-org:mainfrom
Mustafa11300:feat/issue2-agent-test-dsl-revive

Conversation

@Mustafa11300
Copy link
Copy Markdown
Contributor

@Mustafa11300 Mustafa11300 commented Apr 9, 2026

Summary

This PR implements Issue #1599 by adding a declarative Agent Test DSL and Scenario Builder for the testing crate, so agent behavior tests can be defined as reusable scenarios instead of only raw test functions.

Problem

The current testing approach relies mostly on hand-written Rust test functions. That makes multi-turn conversations, tool expectations, and reusable scenario definitions harder to author, review, and maintain.

There is no first-class scenario abstraction for:

  • Multi-turn user and agent interactions
  • Tool call expectations and argument assertions
  • Config-driven scenario authoring for contributors who do not want to write Rust test code directly

What Changed

  • Added a declarative scenario model for agent tests
  • Added a builder-style DSL for multi-turn test authoring
  • Added support for tool-call expectation assertions within scenario steps
  • Added scenario loading support from structured config input
  • Integrated scenario execution with existing reporting flow

Why This Helps

  • Reduces test boilerplate and improves readability
  • Makes behavior tests easier to review as product requirements evolve
  • Improves reusability of scenarios across regression, integration, and CI checks
  • Creates a foundation for future parameterized and golden-response testing

Tests

Following mentor guidance, this PR includes comprehensive tests for the new feature:

  • Unit tests for DSL builder behavior and validation paths
  • Unit tests for scenario parsing and malformed input handling
  • Integration tests for multi-turn execution and tool expectation assertions
  • Regression tests for failure paths and edge cases

All new and existing relevant tests pass.

Practical Examples

Following mentor guidance for structural feature additions, this PR also adds practical end-to-end usage scenarios in examples so new users can understand and adopt the feature quickly.

Examples cover:

  • Basic single-turn scenario definition
  • Multi-turn conversation with tool expectations
  • Config-driven scenario loading and execution
  • Report output and assertion flow in realistic test runs

Verification

Commands used:

  • cargo fmt -p mofa-testing
  • cargo test -p mofa-testing
  • cargo test -p mofa-testing -- --nocapture

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant