FEAT: Add partner integration tests for azure-ai-evaluation red team …#1533
Open
slister1001 wants to merge 4 commits intomicrosoft:mainfrom
Open
FEAT: Add partner integration tests for azure-ai-evaluation red team …#1533slister1001 wants to merge 4 commits intomicrosoft:mainfrom
slister1001 wants to merge 4 commits intomicrosoft:mainfrom
Conversation
…module Add tests/partner_integration/azure_ai_evaluation/ with contract tests validating PyRIT API stability for the azure-ai-evaluation red team module, which depends on 45+ PyRIT imports across 14 files. Test coverage includes: - PromptChatTarget interface contract (extended by 4 SDK classes) - CentralMemory/SQLiteMemory lifecycle (used in RedTeam.__init__) - Data models: Message, MessagePiece, Score, seed models, AttackResult - PromptConverter base + 19 specific converters importability - Scorer/TrueFalseScorer interface (extended by RAIServiceScorer) - Foundry scenario APIs: FoundryScenario, FoundryStrategy, DatasetConfiguration - Exception types and retry decorators - Import smoke tests for azure-ai-evaluation (skipped if not installed) Also adds partner-integration-test target to Makefile. All 84 tests pass with no Azure credentials required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix test_ascii_smuggler_converter_importable to test AsciiSmugglerConverter (was incorrectly testing AsciiArtConverter, duplicating parametrized coverage) - Move module-level asyncio.run(initialize_pyrit_async) to session-scoped fixture - Remove duplicate TestAttackModels (already covered in test_foundry_contract.py) - Extract MinimalTarget to module-level helper (was defined 3x inline) - Add docstring clarifying intentional private API imports in smoke tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| pass | ||
|
|
||
|
|
||
| class TestPromptChatTargetContract: |
Contributor
There was a problem hiding this comment.
jfyi: we're replacing PromptChatTarget with PromptTarget (https://github.com/Azure/PyRIT/pull/1532) that shouldn't affect the result of these tests (and is a minimal change bc PromptChatTarget doesn't offer a whole lot on top of PromptTarget) but this test will change
|
|
||
| def test_foundry_strategy_is_scenario_strategy(self): | ||
| """FoundryStrategy should extend ScenarioStrategy.""" | ||
| from pyrit.scenario import ScenarioStrategy |
Contributor
There was a problem hiding this comment.
why isn't this at the top with DatasetConfiguration ?
Comment on lines
+64
to
+68
| def test_callback_chat_target_extends_prompt_chat_target(self): | ||
| """_CallbackChatTarget must be a subclass of pyrit.prompt_target.PromptChatTarget.""" | ||
| from azure.ai.evaluation.red_team._callback_chat_target import _CallbackChatTarget | ||
|
|
||
| assert issubclass(_CallbackChatTarget, PromptChatTarget) |
Contributor
There was a problem hiding this comment.
these would change too. is there a reason we need the explicit inheritance ?
|
|
||
| def test_seed_prompt_class_exists(self): | ||
| """DatasetConfigurationBuilder creates SeedPrompt instances.""" | ||
| assert SeedPrompt is not None |
Contributor
There was a problem hiding this comment.
i'm assuming there's also a structure you're expecting for the seed models (similar to the message models) ?
…orts, add seed model structural tests - Update PromptChatTarget to PromptTarget per PR microsoft#1532 deprecation - Move ScenarioStrategy import to top-level in test_foundry_contract.py - Add rationale for explicit inheritance checks in test_import_smoke.py - Expand seed model tests with structural validation (value, data_type, harm_categories, role, metadata, SeedGroup composition) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| assert issubclass(FoundryStrategy, ScenarioStrategy) | ||
|
|
||
|
|
||
| class TestFoundryScenarioContract: |
Contributor
There was a problem hiding this comment.
nit: rename to RedTeamScenario
(internally there's a bit of an existential crisis going on where half of the classes are named foundry and the other half is redteam but we're moving to RedTeam so I'd try to use that instead)
hannahwestra25
approved these changes
Mar 26, 2026
Contributor
hannahwestra25
left a comment
hannahwestra25
left a comment
There was a problem hiding this comment.
a few small comments; will set up the pipeline today!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…module
Add tests/partner_integration/azure_ai_evaluation/ with contract tests validating PyRIT API stability for the azure-ai-evaluation red team module, which depends on 45+ PyRIT imports across 14 files.
Test coverage includes:
Also adds partner-integration-test target to Makefile.
All 84 tests pass with no Azure credentials required.
Description
Tests and Documentation