FIX: flaky target capabilities tests causing intermittent failures#1527
Merged
rlundeen2 merged 3 commits intomicrosoft:mainfrom Mar 24, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR stabilizes unit tests around PromptTarget capability resolution by preventing leaked OPENAI_*_UNDERLYING_MODEL environment variables (potentially set by .env loading in other tests) from changing targets from _DEFAULT_CAPABILITIES to _KNOWN_CAPABILITIES.
Changes:
- Added per-test/per-fixture
patch.dict("os.environ", ...)guards to neutralize*_UNDERLYING_MODELenv vars during capability assertions. - Centralized the “clean env” mapping within each affected test module for readability and reuse within that module.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/unit/target/test_target_capabilities.py | Ensures OpenAIVideoTarget modality assertions aren’t affected by leaked underlying-model env vars. |
| tests/unit/target/test_supports_multi_turn.py | Ensures OpenAIVideoTarget supports_multi_turn assertion isn’t flipped by leaked underlying-model env vars. |
| tests/unit/target/test_realtime_target.py | Applies env cleanup at the RealtimeTarget fixture level to keep realtime tests deterministic. |
rlundeen2
changed the title
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add OPENAI_COMPLETION_UNDERLYING_MODEL and OPENAI_RESPONSES_UNDERLYING_MODEL to _CLEAN_UNDERLYING_MODEL_ENV in test_target_capabilities.py - Apply @patch.dict to all OpenAI target test methods in TestTargetCapabilitiesModalities (chat, image, tts, realtime, response, completion) - not just video - Add @patch.dict to TTS and completion tests in test_supports_multi_turn.py - Add TTS and completion env vars to test_supports_multi_turn.py dict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hannahwestra25
approved these changes
Mar 23, 2026
jsong468
approved these changes
Mar 23, 2026
jbolor21
pushed a commit
to jbolor21/jbolor-PyRIT
that referenced
this pull request
Mar 25, 2026
…icrosoft#1527) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The env vars would get polluted when running in parallel, causing test failures