There are three ways users primarily interact with PyRIT: The GUI, CLI, and framework. Although we have been chipping away at these for a while, we are at a state where all three of these interactions are ready for use.
The framework is by far the most mature, but in the next few releases you'll see these coalescing.
pyrit_scan and pyrit_shell (CLI)
Scenarios are a way to point PyRIT at a target and run assessments via the command line.
They are still new, so there are not a ton of existing scenarios, but we expect this list to grow. This release adds the following:
| Scenario | Family | What it tests |
|---|---|---|
| Scam | AIRT | Generating phishing/fraud material via persuasion techniques (single/multi-turn) |
| Leakage | AIRT | Susceptibility to leaking PII, IP, credentials, secrets (single/multi-turn, image-based, Crescendo) |
| Psychosocial | AIRT | Harmful psychosocial behavior — mishandling crises, impersonating therapists |
| Jailbreak | AIRT | Vulnerability to jailbreak attacks: PromptSending, ManyShot, SkeletonKey, RolePlay (simple vs complex) |
| RedTeamAgent | Foundry | Preconfigured multi-difficulty red-teaming with 25+ attack strategies across easy/moderate/difficult |
We added a new YAML-based configuration system loaded from ~/.pyrit/.pyrit_conf which configures memory, targets, scorers and other default properties.
There were several improvements to the CLI tools themselves including more intuitive options and better error handling. Most importantly, we added an Animated ASCII raccoon banner on startup for pyrit_shell
CoPyRIT (GUI)
CoPyRIT brings the power of PyRIT into an easy interface. This should be considered pre-release at the moment, but it is already useful and being used by our Microsoft AI Red Team. This release is the first time the GUI is usable (at your own risk).
Framework
⚠️ Breaking Changes
- Registry tags:
RegistryEntry[T]wrapper type and tag support for instance registries, enabling categorization and tag-based retrieval of registered components (#1485) TAPadded to content harms scenario with restructured scenario configuration (#1378)- Dataset loading refactor: New
SeedDatasetProviderandSeedDatasetFilterfor flexible dataset discovery and filtering (#1451) - CLI parameter changes: Replaced
openai_objective_targetinitializer with explicit--targetCLI parameter (#1536) - Registry-based default objective scorer in scenarios (#1528)
Please review the deprecation notes and migration guidance before upgrading.
🎯 Targets
TargetRegistryand AIRT Targets Initializer, allowing central management of targets (#1320)- Added
TargetCapabilities, paving the way for us to better make decisions depending on what a target is capable of (#1433, #1464) - Added reasoning_effort and reasoning_summary to
OpenAIResponseTarget(#1385) OpenAIVideoTarget: support remix, image-to-video (#1341)- Added
image_pathdata type support inWebSocketCopilotTarget(#1345)
📚 Datasets
- New
SeedDatasetProviderandSeedDatasetFilterfor flexible dataset loading (#1451) - Added 8 new dataset loaders: CBT-Bench (#1411), PromptIntel (#1400), SimpleSafetyTests (#1426), SALAD-Bench (#1425), OR-Bench (#1423), BeaverTails (#1424), ToxicChat (#1422), HarmfulQA (#1421)
🔄 Converters
- Added
WordDocConverterfor Word document prompt injection (#1368) - Added
JsonStringConverterfor JSON-safe string conversion (#1347) - Added
ScientificTranslationConverter(#1379) - Audio converters for speed, white noise, echo, and volume (#1375)
- Generalized
ColloquialWordswapConverter(#1348)
📊 Scoring
- Added Scorer evaluations with metrics tracking, giving us data to improve scoring accuracy (#1455, #1406)
- Registry-based scorers allowing for centralized scorer management (#1528)
- Audio scoring support (#1337)
- Flexible scale
LikertScorer(#1444) - Improvements to
SelfAskRefulsalScorer(#1366) - Improvements to
SelfAskTrueFalseScorer(#1342) - Added scorer support for reasoning targets (#1403)
🐛Multiple bug Fixes and Improvements
New Contributors
- @akashWhoCodes made their first contribution in #1347
- @nmolivo made their first contribution in #1356
- @mhadica made their first contribution in #1368
- @anandansundar made their first contribution in #1400
- @taherakolawala made their first contribution in #1348
- @spencrr made their first contribution in #1409
- @warisgill made their first contribution in #1411
- @biefan made their first contribution in #1469
- @behnam-o made their first contribution in #1538
Full list of changes
- FEAT Use TASK_ACHIEVED as fallback for SelfAskTrueFalseScorer by @fdubut in #1342
- FEAT: Jailbreak Scenario by @ValbuenaVC in #1329
- FEAT: Add JsonStringConverter for JSON-safe string conversion (#454) by @akashWhoCodes in #1347
- FIX MS Package Signing Key issue #1353 to enable DevContainers after Feb-01-2026 by @nmolivo in #1356
- MAINT address code scanning alert on url sanitization by @romanlutz in #1361
- MAINT bump package versions to address dependabot alerts by @romanlutz in #1360
- MAINT mypy fixes by @romanlutz in #1359
- MAINT upgrade vite in response to npm audit by @romanlutz in #1358
- FEAT: Target Registry and AIRT Targets Initializer by @jsong468 in #1320
- MAINT address remaining dependabout issues with uv.lock by @romanlutz in #1363
- FEAT local Docker setup for running GUI or Jupyter by @romanlutz in #1357
- FEAT: Adding audio scoring by @jbolor21 in #1337
- MAINT: PEP 8: Uppercase class constants (VERSION, _DEFAULT_VALIDATOR) by @Copilot in #1355
- FEAT: PyRIT Config by @ValbuenaVC in #1343
- FEAT: Attack Identifier by @rlundeen2 in #1364
- FEAT Add backend APIs by @romanlutz in #1354
- Decoupled ScenarioMetadata and InitializerMetadata from Identity by introducing RegistryEntry by @bashirpartovi in #1370
- FIX Support errors in
MultiPromptSendingAttack, add safe completion support toSelfAskRefusalScorerby @fdubut in #1366 - FIX: OPENAI_VIDEO environment variable fix by @jsong468 in #1376
- MAINT: .pyrit_conf Docs by @ValbuenaVC in #1374
- FEAT: Sora target: support remix, image-to-video by @varunj-msft in #1341
- FEAT: Jailbreak Scenario Expansion by @ValbuenaVC in #1340
- FEAT Add WordDocConverter by @mhadica in #1368
- MAINT: upgrade dependencies for Dependabot security alerts by @romanlutz in #1382
- MAINT Clean up GCG code style and add unit tests by @romanlutz in #1380
- FEAT: add support for
image_pathdata type inWebSocketCopilotTargetby @paulinek13 in #1345 - MAINT Enable ruff W (pycodestyle warnings) rules and fix all violations by @romanlutz in #1391
- MAINT Enable ruff C4 (flake8-comprehensions) rules and fix all violations by @romanlutz in #1392
- MAINT Migrating from the old Identifier to ComponentIdentifier to reduce coupling and simplify the design by @bashirpartovi in #1387
- MAINT Speed up jailbreak unit tests from 3:42m to 33s by @bashirpartovi in #1399
- FEAT: add reasoning_effort and reasoning_summary to OpenAIResponseTarget by @romanlutz in #1385
- MAINT Deprecate
HumanInTheLoopScorerGradioandHumanInTheLoopConverterin favor of new GUI by @romanlutz in #1386 - DOC Complete ruff docstring cleanup for exceptions and models by @romanlutz in #1388
- MAINT Enable ruff RET (flake8-return) rules and fix all violations by @romanlutz in #1396
- FEAT: Audio Converters for speed, white noise, echo, and volume by @petebryan in #1375
- FEAT: Adding SeedAttackTechniqueGroup by @rlundeen2 in #1373
- MAINT Enable ruff UP (pyupgrade) rules and fix all violations by @romanlutz in #1389
- MAINT Enable ruff SIM (flake8-simplify) rules and fix all violations by @romanlutz in #1397
- MAINT Regenerate uv.lock for dependabot security fixes by @romanlutz in #1401
- FIX Mock tokenizer in unit test to avoid HuggingFace network call by @romanlutz in #1402
- FIX: Ensuring
_score_value_with_llmworks with reasoning targets by @jsong468 in #1403 - FEAT Add PromptIntel remote dataset loader by @anandansundar in #1400
- MAINT Update cryptography>=46.0.5 and werkzeug>=3.1.6 for dependabot alerts by @romanlutz in #1405
- MAINT Enable ruff PIE (flake8-pie) rules and fix all violations by @romanlutz in #1394
- MAINT Bump rollup to >=4.59.0 for dependabot security alert by @romanlutz in #1408
- MAINT Enable ruff B (flake8-bugbear) rules and fix all violations by @romanlutz in #1390
- MAINT Enable ruff TCH (flake8-type-checking) rules and fix all violations by @romanlutz in #1395
- MAINT add MacOS CI by @romanlutz in #1407
- FEAT: Generalize Colloquial Wordswap Attack Converter by @taherakolawala in #1348
- FIX circular import making pyrit.version unavailable during submodule init by @Copilot in #1428
- FEAT Add SimpleSafetyTests dataset loader by @romanlutz in #1426
- MAINT Migrate Azure Cognitive Services from API key to Entra ID authentication by @romanlutz in #1404
- FEAT Expand memory interface and models for attack results by @romanlutz in #1420
- FEAT Add SALAD-Bench dataset loader by @romanlutz in #1425
- MAINT Enable ruff rule E (pycodestyle errors) by @romanlutz in #1413
- MAINT Enable ruff PGH rule for pygrep-hooks linting by @romanlutz in #1416
- FEAT Add OR-Bench dataset loader by @romanlutz in #1423
- MAINT Enable ruff PERF rule for performance linting by @romanlutz in #1415
- FIX address deprecation warnings by @romanlutz in #1430
- MAINT Enable ruff rules: DTZ, N, T10, TID, YTT by @romanlutz in #1414
- FIX replace api key in scenarios by @hannahwestra25 in #1435
- MAINT Add pre-commit hook to sanitize user paths in notebook outputs by @romanlutz in #1429
- MAINT Enable low-hanging ruff rules: A, ERA, EXE, FA, FLY, ICN, ISC, LOG, PLE, Q, RSE, SLOT by @romanlutz in #1412
- MAINT/FIX: Generalizing eval-hash by @rlundeen2 in #1434
- FIX: Add arm64 Support for
devcontainerby @spencrr in #1409 - FEAT Add BeaverTails dataset loader by @romanlutz in #1424
- FEAT Add TargetCapabilities with supports_multi_turn and adapt attacks accordingly by @romanlutz in #1433
- [BREAKING] FEAT add TAP to content harms scenario by @hannahwestra25 in #1378
- MAINT set up GitHub Copilot in devcontainer by @romanlutz in #1431
- FEAT Add ToxicChat dataset loader by @romanlutz in #1422
- FEAT: Add new human labeled datasets by @jsong468 in #1436
- FEAT Add HarmfulQA dataset loader by @romanlutz in #1421
- FEAT: Scientific Translation Converter by @jbolor21 in #1379
- MAINT: Add permissions to docker_build workflow by @spencrr in #1441
- MAINT: Bump pip deps by @spencrr in #1442
- TEST: add unit tests for ConverterRegistry by @spencrr in #1440
- FEAT: Flexible Scale Likert Scoring by @jsong468 in #1444
- FEAT Backend attack API: conversation-centric redesign with multi-conversation workspaces and media serving by @romanlutz in #1419
- MAINT Updating Release Instructions by @ValbuenaVC in #1449
- FEAT: atomic attack identifier by @rlundeen2 in #1446
- FEAT: Update evaluate_scorers by @varunj-msft in #1406
- FIX: Reorder scorer metrics notebook in table of contents by @jsong468 in #1452
- FIX: Fixing SQL Azure Integration Tests by @rlundeen2 in #1457
- MAINT: Adding Scorer Evals by @rlundeen2 in #1455
- MAINT Fix integration test import errors and runtime issues by @romanlutz in #1448
- DOC: Add Release Readiness step to release process docs by @rlundeen2 in #1450
- FIX use cognitiveservices scope for all Azure AI endpoints by @romanlutz in #1453
- FEAT Wire frontend attack view to backend APIs by @romanlutz in #1371
- Fix type annotation warnings and test warnings (issue #442) by @romanlutz in #1459
- FIX address dependabot alerts by bumping package versions by @romanlutz in #1460
- FIX: Adding openai invalid_prompt safety blocks as content filters by @rlundeen2 in #1463
- FEAT Animated ASCII banner with raccoon mascot for PyRIT CLI by @romanlutz in #1417
- FEAT: CBT-Bench Dataset by @warisgill in #1411
- DOC Upgrade to jupyterbook v2 and add proper landing page by @romanlutz in #1458
- DOC GitHub Pages 404: use static HTML output for deployment by @romanlutz in #1465
- DOC fix pages deploy by @romanlutz in #1466
- FIX: fixing navbar link by @rlundeen2 in #1468
- DOC improve API reference accuracy, structure, and readability by @biefan in #1469
- Fix multiline JSON extraction in exceptions helpers by @biefan in #1474
- Fix explicit SAS token handling in AzureBlobStorageIO by @biefan in #1473
- FEAT: Adding PyRITInitializer parameters by @rlundeen2 in #1456
- DOC: Add bibliography support with BibTeX citations across documentation by @romanlutz in #1472
- FEAT AzureContentFilterScorer: Switch to async client and accept async auth providers by @adrian-gavrila in #1467
- Preserve URL case in HTTP target requests by @biefan in #1484
- FEAT: Capture token usage from ChatCompletion response in OpenAIChatTarget by @slister1001 in #1476
- DOC: updating copilot review instructions by @rlundeen2 in #1477
- MAINT: Removing pydub as a dependency by @jbolor21 in #1445
- Support CRLF raw HTTP requests in HTTPTarget by @biefan in #1491
- [BUG] Fix JSON path for converter class names in attack result queries by @jbolor21 in #1512
- FIX GUI promote conversation to main feature working by @adrian-gavrila in #1513
- Preserve empty JSON schema metadata by @biefan in #1488
- Ignore blank lines when reading TXT prompts by @biefan in #1480
- Ignore blank lines when reading JSONL by @biefan in #1479
- FIX GUI conversation switching during in-flight requests and sort ordering by @adrian-gavrila in #1517
- Handle zero tail slices in SeedDataset.get_values by @biefan in #1511
- FIX Preserve silent when loading config overrides by @biefan in #1500
- FIX Reject empty WMDP category values by @biefan in #1497
- FEAT expand TargetCapabilities by @hannahwestra25 in #1464
- FIX: PyRITShell startup deadlock and improve shell startup time by @biefan in #1489
- FEAT: Dataset Loading Changes by @ValbuenaVC in #1451
- FEAT Breaking: Adding tags to registry classes by @rlundeen2 in #1485
- FIX align platform oai key by @hannahwestra25 in #1522
- FIX missing custom capabilities in integration test by @hannahwestra25 in #1521
- FIX: Small fixes to CLI docs and openai_objective_target initializer by @jsong468 in #1524
- Preserve request params and validate upload files in HTTPXAPITarget by @biefan in #1487
- Ignore imported initializer classes in script discovery by @biefan in #1509
- Fix: Eval hash mismatch due to parameter truncation in DB storage by @rlundeen2 in #1523
- MAINT: Optimize devcontainer Dockerfile by @spencrr in #1437
- Preserve empty URL query parameters by @biefan in #1494
- FIX: flaky target capabilities tests causing intermittent failures by @rlundeen2 in #1527
- FEAT: Use registry-based default objective scorer in scenarios by @rlundeen2 in #1528
- FIX pre-commit http error by @hannahwestra25 in #1534
- FEAT: Replace openai_objective_target initializer with explicit --target CLI parameter by @rlundeen2 in #1536
- FIX: Fixes for failing integration tests by @varunj-msft in #1539
- MAINT: Use Composition instead of Multi-Inheritance in Video Scorers by @behnam-o in #1538
- FIX: Fixed fairness_bias.yaml in self_ask_likert_scorer.py and metrics for evaluate_scorers by @varunj-msft in #1541
- FIX: Update E2E tests by @rlundeen2 in #1542
- DOC: Updating Getting Started Docs by @rlundeen2 in #1540
- FIX: Add *.jsonl to MANIFEST.in for scorer metrics packaging by @varunj-msft in #1546
📜 Full Changelog
Contributors
bashirpartovi, anandansundar, and 20 other contributors