Skip to content

feat(testing): Add Parameterized Test Support with Test Data Generator#1606

Open
Mustafa11300 wants to merge 2 commits intomofa-org:mainfrom
Mustafa11300:feat/issue3-parameterized-test-support
Open

feat(testing): Add Parameterized Test Support with Test Data Generator#1606
Mustafa11300 wants to merge 2 commits intomofa-org:mainfrom
Mustafa11300:feat/issue3-parameterized-test-support

Conversation

@Mustafa11300
Copy link
Copy Markdown
Contributor

🔍 Description

Adds parameterized scenario expansion to the mofa-testing crate. A single scenario template can now expand into many concrete test cases by substituting {{variable}} placeholders with values from parameter sets.

This is Issue 3 from the testing platform roadmap (Test Definition Layer, Phase 2).

Closes #1603
Depends on #1599

📌 Changes

New: tests/src/parameterized.rs

Core parameterized test module with the following public components:

Component Purpose
ParameterSet Named map of variable → value bindings for one test variant
ParameterMatrix Cartesian product expansion with configurable safety limits
ParameterizedScenario Wraps an AgentTestScenario template and expands it against parameter sets
ParameterizedScenarioFile YAML/TOML/JSON file-backed parameterized scenario loading
ParameterExpansionError Structured error type for all expansion failure modes

Placeholder substitution ({{variable}}) is supported in:

  • user_input (turn prompts)
  • system_prompt
  • text expectations (respond_containing, respond_exact)
  • pattern expectations (respond_matching_regex)
  • name expectations (call_tool, call_tool_with)
  • arguments values (recursively in JSON)

Expansion modes:

  • Explicit parameter list: named sets with direct variable bindings
  • Matrix expansion: Cartesian product of variable dimensions
  • Mixed: both explicit and matrix sets combined

Modified: tests/src/lib.rs

  • Added pub mod parameterized module registration
  • Added public re-exports: ParameterExpansionError, ParameterMatrix, ParameterSet, ParameterizedScenario, ParameterizedScenarioFile

New: tests/tests/parameterized_tests.rs

30+ comprehensive tests covering:

  • ParameterSet: builder, variable ordering, serialization roundtrip
  • ParameterMatrix: 1/2/3 dimensions, combination count, empty dimension error, no dimensions error, limit exceeded error
  • Expansion: placeholder substitution in all field types (user_input, system_prompt, respond_containing, respond_exact, respond_matching_regex, call_tool_with arguments)
  • Execution: expanded scenarios run with agents and produce correct reports; failure detection in parameterized variants
  • File loading: YAML, TOML, JSON parameterized scenario files; matrix expansion from files; mixed explicit + matrix; file-level limit enforcement
  • Edge cases: no placeholders, multiple placeholders in single field, same placeholder used multiple times, missing variable detection, empty parameter sets, stable naming

New: examples/parameterized_test/

File Description
README.md Usage guide with code samples and placeholder syntax reference
scenario_weather_parameterized.yaml Explicit parameter list: weather lookup across 3 cities
scenario_greetings_matrix.yaml Matrix expansion: 2 languages × 3 tones = 6 variants
scenario_support_parameterized.toml TOML: parameterized support ticket lookup across departments

🧪 Testing

All new functionality is covered by tests/tests/parameterized_tests.rs with 30+ test cases.

Key test categories:

  1. Unit tests — ParameterSet and ParameterMatrix in isolation
  2. Integration tests — Full expansion + scenario execution with ScriptedAgent
  3. File loading tests — YAML, TOML, JSON deserialization and expansion
  4. Error path tests — Empty sets, missing variables, matrix limit exceeded, empty dimensions
  5. Edge case tests — No placeholders, multiple placeholders, stable naming, serialization

💡 Usage Example

Builder API (Rust)

use mofa_testing::{AgentTest, ParameterSet, ParameterMatrix, ParameterizedScenario};

// 1. Define a template with {{variable}} placeholders
let template = AgentTest::new("weather_agent")
    .given_tool("weather_search")
    .when_user_says("What's the weather in {{city}}?")
    .then_agent_should()
    .call_tool("weather_search")
    .respond_containing("{{city}}")
    .build()?;

// 2. Define parameter sets (explicit or matrix)
let sets = ParameterMatrix::new()
    .dimension("city", vec!["Berlin", "Tokyo", "NYC"])
    .expand()?;

// 3. Expand and run
let parameterized = ParameterizedScenario::new(template, sets);
let expanded = parameterized.expand()?; // 3 concrete scenarios

for scenario in &expanded {
    let report = scenario.run_with_agent(&mut agent).await;
    println!("[{}] passed={}", scenario.agent_id, report.passed());
}

File-backed (YAML)

template:
  agent_id: weather_agent
  tools: [weather_search]
  turns:
    - user: "What's the weather in {{city}}?"
      expect:
        - kind: call_tool
          name: weather_search
        - kind: respond_containing
          text: "{{city}}"

parameters:
  - name: berlin
    vars: { city: Berlin }
  - name: tokyo
    vars: { city: Tokyo }
let yaml = std::fs::read_to_string("scenario.yaml")?;
let parameterized = ParameterizedScenarioFile::from_yaml_str(&yaml)?;
let expanded = parameterized.expand()?;

✅ Checklist

  • Code follows project conventions and style
  • New module registered in lib.rs with public re-exports
  • Comprehensive tests added (parameterized_tests.rs)
  • Examples added (examples/parameterized_test/)
  • README with usage documentation included
  • Error handling with structured error types
  • No breaking changes to existing APIs
  • Builds on the DSL foundation from feat(testing): Add Agent Test DSL with Declarative Scenario Builder #1599

Implements parameterized scenario expansion for the mofa-testing crate.
One scenario template can now expand into many concrete test cases by
substituting {{variable}} placeholders with values from parameter sets.

New components:
- ParameterSet: named variable bindings for one test variant
- ParameterMatrix: Cartesian product expansion with safety limits
- ParameterizedScenario: template + parameter sets -> expanded scenarios
- ParameterizedScenarioFile: YAML/TOML/JSON file-backed loading

Includes:
- 30+ comprehensive tests covering expansion, substitution, file
  loading, execution, edge cases, and error handling
- Example scenarios in examples/parameterized_test/ with YAML, TOML,
  and matrix expansion demonstrations
- README with usage guide and code samples

Closes #<ISSUE_NUMBER>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(testing): Add Parameterized Test Support with Test Data Generator

1 participant