Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
3390089
added first iteration of the cot hijacking attack strategy
riyosha Mar 5, 2026
7a0e340
FEAT Add HarmfulQA dataset loader (#1421)
romanlutz Mar 5, 2026
5179e46
FEAT: Scientific Translation Converter (#1379)
jbolor21 Mar 5, 2026
06d8432
MAINT: Add permissions to docker_build workflow (#1441)
spencrr Mar 7, 2026
c18ea4d
MAINT: Bump pip deps (#1442)
spencrr Mar 9, 2026
80dc9ef
TEST: add unit tests for ConverterRegistry (#1440)
spencrr Mar 9, 2026
c1a9def
FEAT: Flexible Scale Likert Scoring (#1444)
jsong468 Mar 9, 2026
2e562b5
FEAT Backend attack API: conversation-centric redesign with multi-con…
romanlutz Mar 9, 2026
05963c4
MAINT Updating Release Instructions (#1449)
ValbuenaVC Mar 10, 2026
1c480a9
FEAT: atomic attack identifier (#1446)
rlundeen2 Mar 10, 2026
361f693
FEAT: Update evaluate_scorers (#1406)
varunj-msft Mar 10, 2026
1537689
FIX: Reorder scorer metrics notebook in table of contents (#1452)
jsong468 Mar 11, 2026
d5284cb
FIX: Fixing SQL Azure Integration Tests (#1457)
rlundeen2 Mar 11, 2026
418169f
MAINT: Adding Scorer Evals (#1455)
rlundeen2 Mar 11, 2026
75f8e2f
MAINT Fix integration test import errors and runtime issues (#1448)
romanlutz Mar 12, 2026
060b17a
DOC: Add Release Readiness step to release process docs (#1450)
rlundeen2 Mar 13, 2026
9a7461f
FIX use cognitiveservices scope for all Azure AI endpoints (#1453)
romanlutz Mar 13, 2026
aa5623d
FEAT Wire frontend attack view to backend APIs (#1371)
romanlutz Mar 13, 2026
15e7149
Fix type annotation warnings and test warnings (issue #442) (#1459)
romanlutz Mar 13, 2026
df6b9b5
FIX address dependabot alerts by bumping package versions (#1460)
romanlutz Mar 13, 2026
41f0aa0
FIX: Adding openai invalid_prompt safety blocks as content filters (#…
rlundeen2 Mar 13, 2026
b04f1ad
FEAT Animated ASCII banner with raccoon mascot for PyRIT CLI (#1417)
romanlutz Mar 13, 2026
9536967
FEAT: CBT-Bench Dataset (#1411)
Mar 13, 2026
819c49c
DOC Upgrade to jupyterbook v2 and add proper landing page (#1458)
romanlutz Mar 13, 2026
bbcfe9e
DOC GitHub Pages 404: use static HTML output for deployment (#1465)
romanlutz Mar 13, 2026
2d65fe4
DOC fix pages deploy (#1466)
romanlutz Mar 13, 2026
7e3706d
FIX: fixing navbar link (#1468)
rlundeen2 Mar 14, 2026
b6e1896
DOC generate API pages for empty modules and render aliases (#1469)
biefan Mar 16, 2026
32bae1c
Fix multiline JSON extraction in exceptions helpers (#1474)
biefan Mar 16, 2026
28ccaa5
Fix explicit SAS token handling in AzureBlobStorageIO (#1473)
biefan Mar 16, 2026
3909aee
FEAT: Adding PyRITInitializer parameters (#1456)
rlundeen2 Mar 16, 2026
a0ae027
DOC: Add bibliography support with BibTeX citations across documentat…
romanlutz Mar 16, 2026
66ab015
FEAT AzureContentFilterScorer: Switch to async client and accept asyn…
adrian-gavrila Mar 17, 2026
97056a8
Preserve URL case in HTTP target requests (#1484)
biefan Mar 17, 2026
0ff9ef3
FEAT: Capture token usage from ChatCompletion response in OpenAIChatT…
slister1001 Mar 17, 2026
cf7b9dd
DOC: updating copilot review instructions (#1477)
rlundeen2 Mar 17, 2026
96357b0
MAINT: Removing pydub as a dependency (#1445)
jbolor21 Mar 17, 2026
e421751
Support CRLF raw HTTP requests in HTTPTarget (#1491)
biefan Mar 18, 2026
37bcdaa
[BUG] Fix JSON path for converter class names in attack result querie…
jbolor21 Mar 18, 2026
707d93a
FIX GUI promote conversation to main feature working (#1513)
adrian-gavrila Mar 18, 2026
9173b62
Preserve empty JSON schema metadata (#1488)
biefan Mar 18, 2026
8e90556
Ignore blank lines when reading TXT prompts (#1480)
biefan Mar 18, 2026
21664dd
Ignore blank lines when reading JSONL (#1479)
biefan Mar 18, 2026
756280a
FIX GUI conversation switching during in-flight requests and sort ord…
adrian-gavrila Mar 18, 2026
3a9d8bc
Handle zero tail slices in SeedDataset.get_values (#1511)
biefan Mar 19, 2026
df1ad7a
FIX Preserve silent when loading config overrides (#1500)
biefan Mar 19, 2026
b77b367
FIX Reject empty WMDP category values (#1497)
biefan Mar 19, 2026
21f7a9f
FEAT expand TargetCapabilities (#1464)
hannahwestra25 Mar 19, 2026
9f51e0c
FIX: PyRITShell startup deadlock and improve shell startup time (#1489)
biefan Mar 19, 2026
640a8ba
FEAT: Dataset Loading Changes (#1451)
ValbuenaVC Mar 19, 2026
420c91c
FEAT Breaking: Adding tags to registry classes (#1485)
rlundeen2 Mar 19, 2026
2d0def7
FIX align platform oai key (#1522)
hannahwestra25 Mar 20, 2026
7749d7c
FIX missing custom capabilities in integration test (#1521)
hannahwestra25 Mar 20, 2026
fc56019
FIX: Small fixes to CLI docs and openai_objective_target initializer …
jsong468 Mar 20, 2026
7a68447
Preserve request params and validate upload files in HTTPXAPITarget (…
biefan Mar 20, 2026
38273e7
Ignore imported initializer classes in script discovery (#1509)
biefan Mar 20, 2026
9465aba
Fix: Eval hash mismatch due to parameter truncation in DB storage (#…
rlundeen2 Mar 20, 2026
157ebd6
MAINT: Optimize devcontainer Dockerfile (#1437)
spencrr Mar 21, 2026
4664680
incorporated PR feedback
riyosha Mar 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
83 changes: 40 additions & 43 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
FROM mcr.microsoft.com/devcontainers/python:3.11
# syntax=docker/dockerfile:1
FROM mcr.microsoft.com/devcontainers/python:3.11-bookworm

# Makes installation faster
ENV UV_COMPILE_BYTECODE=1
ENV DEBIAN_FRONTEND=noninteractive

SHELL ["/bin/bash", "-c"]

Expand All @@ -11,66 +13,61 @@ USER root
RUN rm -f /etc/apt/sources.list.d/yarn.list 2>/dev/null || true

# Install required system packages + ODBC prerequisites
RUN apt-get update && apt-get install -y \
sudo \
unixodbc \
unixodbc-dev \
libgl1 \
git \
curl \
xdg-utils \
build-essential \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
apt-get update \
&& apt-get install -y --no-install-recommends \
sudo \
unixodbc \
unixodbc-dev \
libgl1 \
git \
curl \
xdg-utils \
build-essential

# Install the Azure CLI, Microsoft ODBC Driver 18 & SQL tools
# Install Microsoft ODBC Driver 18 & SQL tools
# Note: Debian Trixie's sqv rejects SHA1 signatures, so we use gpg directly to import the Microsoft key
RUN apt-get update && apt-get install -y \
apt-transport-https \
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
gnupg \
lsb-release \
&& curl -sL https://packages.microsoft.com/keys/microsoft.asc \
| gpg --dearmor \
> /usr/share/keyrings/microsoft-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/microsoft-archive-keyring.gpg] https://packages.microsoft.com/debian/12/prod bookworm main" \
> /etc/apt/sources.list.d/microsoft.list \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql18 mssql-tools18 \
&& apt-get install -y azure-cli \
&& echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> /etc/profile.d/sqltools.sh \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
&& ACCEPT_EULA=Y apt-get install -y --no-install-recommends \
msodbcsql18 \
mssql-tools18 \
&& echo 'export PATH="$PATH:/opt/mssql-tools18/bin"' >> /etc/profile.d/sqltools.sh

# audio back-ends needed by Azure Speech SDK
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install -y --no-install-recommends \
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
apt-get update \
&& apt-get install -y --no-install-recommends \
libasound2 \
libpulse0 \
&& rm -rf /var/lib/apt/lists/*
libpulse0

# Install uv system-wide and create pyrit-dev venv
RUN curl -LsSf https://astral.sh/uv/install.sh | sh \
&& mv /root/.local/bin/uv /usr/local/bin/uv \
&& rm -rf /opt/venv \
&& uv venv /opt/venv --python 3.11 --prompt pyrit-dev \
&& chown -R vscode:vscode /opt/venv \
&& ls -la /opt/venv/bin/activate
COPY --from=ghcr.io/astral-sh/uv:0.10.8 /uv /uvx /bin/
RUN uv venv /opt/venv --python 3.11 --prompt pyrit-dev \
&& chown -R vscode:vscode /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install Node.js 24.x LTS for frontend development
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt/lists,sharing=locked \
curl -fsSL https://deb.nodesource.com/setup_24.x | bash - \
&& apt-get install -y --no-install-recommends nodejs

# vscode user already exists in the base image, just ensure sudo access
RUN echo "vscode ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

# Install Node.js 20.x and npm for frontend development
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs \
&& npm install -g npm@latest \
&& npm install -g @github/copilot@0.0.421 \
&& npm cache clean --force \
&& rm -rf /root/.npm \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

# Pre-create common user caches and fix permissions
RUN mkdir -p /home/vscode/.cache/pre-commit \
&& mkdir -p /home/vscode/.vscode-server \
Expand All @@ -79,7 +76,7 @@ RUN mkdir -p /home/vscode/.cache/pre-commit \
&& mkdir -p /home/vscode/.cache/venv \
&& mkdir -p /home/vscode/.cache/pylance \
&& chown -R vscode:vscode /home/vscode/.cache /home/vscode/.vscode-server \
&& chmod -R 777 /home/vscode/.cache/pip /home/vscode/.cache/pylance /home/vscode/.cache/venv /home/vscode/.cache/uv\
&& chmod -R 755 /home/vscode/.cache/pip /home/vscode/.cache/pylance /home/vscode/.cache/venv /home/vscode/.cache/uv \
&& chmod -R 755 /home/vscode/.vscode-server

USER vscode
Expand All @@ -95,6 +92,6 @@ RUN git config --global core.preloadindex true \
&& git config --global status.showUntrackedFiles all \
&& git config --global core.fsmonitor true

# Set cache directories so they can be mounted
# Set cache directories so they can be mounted
ENV PIP_CACHE_DIR="/home/vscode/.cache/pip"
ENV UV_CACHE_DIR="/home/vscode/.cache/uv"
8 changes: 8 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,14 @@
]
}
},
"features": {
"ghcr.io/devcontainers/features/azure-cli:1": {
"version": "latest"
},
"ghcr.io/devcontainers/features/copilot-cli:1": {
"version": "latest"
}
},
"postCreateCommand": "/bin/bash -i .devcontainer/devcontainer_setup.sh",
"forwardPorts": [3000, 4213, 5000, 8000, 8888]
}
13 changes: 9 additions & 4 deletions .devcontainer/devcontainer_setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ if [ ! -d "$MYPY_CACHE" ]; then
echo "Creating mypy cache directory..."
sudo mkdir -p $MYPY_CACHE
sudo chown vscode:vscode $MYPY_CACHE
sudo chmod 777 $MYPY_CACHE
sudo chmod 755 $MYPY_CACHE
else
# Check ownership
OWNER=$(stat -c '%U:%G' $MYPY_CACHE)
Expand All @@ -21,9 +21,9 @@ else
# Check permissions
PERMS=$(stat -c '%a' $MYPY_CACHE)

if [ "$PERMS" != "777" ]; then
if [ "$PERMS" != "755" ]; then
echo "Fixing mypy cache directory permissions..."
sudo chmod -R 777 $MYPY_CACHE
sudo chmod -R 755 $MYPY_CACHE
fi
fi

Expand Down Expand Up @@ -70,6 +70,7 @@ if [ -f "package.json" ]; then
npm install

# Install Playwright browsers and system dependencies for E2E testing
# This may fail if apt repos have signature issues - don't block setup
echo "📦 Installing Playwright browsers..."

# Remove third-party repos with SHA1 signature issues (rejected since 2026-02-01)
Expand All @@ -78,7 +79,11 @@ if [ -f "package.json" ]; then
/etc/apt/sources.list.d/nodesource.list \
/etc/apt/sources.list.d/microsoft.list 2>/dev/null || true

npx playwright install --with-deps chromium
if npx playwright install --with-deps chromium; then
echo "✅ Playwright browsers installed."
else
echo "⚠️ Playwright installation failed (apt signature issues). Run 'npx playwright install chromium' manually if needed for E2E tests."
fi

echo "✅ Frontend dependencies installed."
fi
Expand Down
4 changes: 2 additions & 2 deletions .env_example
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
###################################

PLATFORM_OPENAI_CHAT_ENDPOINT="https://api.openai.com/v1"
PLATFORM_OPENAI_CHAT_API_KEY="sk-xxxxx"
PLATFORM_OPENAI_CHAT_KEY="sk-xxxxx"
PLATFORM_OPENAI_CHAT_GPT4O_MODEL="gpt-4o"

# Note: For Azure OpenAI endpoints, use the new format with /openai/v1 and specify the model separately
Expand Down Expand Up @@ -79,7 +79,7 @@ DEFAULT_OPENAI_FRONTEND_KEY = ${AZURE_OPENAI_GPT4O_AAD_KEY}
DEFAULT_OPENAI_FRONTEND_MODEL = "gpt-4o"

OPENAI_CHAT_ENDPOINT=${PLATFORM_OPENAI_CHAT_ENDPOINT}
OPENAI_CHAT_KEY=${PLATFORM_OPENAI_CHAT_API_KEY}
OPENAI_CHAT_KEY=${PLATFORM_OPENAI_CHAT_KEY}
OPENAI_CHAT_MODEL=${PLATFORM_OPENAI_CHAT_GPT4O_MODEL}
# The following line can be populated if using an Azure OpenAI deployment
# where the deployment name differs from the actual underlying model
Expand Down
2 changes: 1 addition & 1 deletion .env_local_example
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

# This will override the .env value for your default OpenAIChatTarget
OPENAI_CHAT_ENDPOINT=${PLATFORM_OPENAI_CHAT_ENDPOINT}
OPENAI_CHAT_KEY=${PLATFORM_OPENAI_CHAT_API_KEY}
OPENAI_CHAT_KEY=${PLATFORM_OPENAI_CHAT_KEY}
OPENAI_CHAT_MODEL="gpt-4o"


Expand Down
31 changes: 31 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# PyRIT - Repository Instructions

PyRIT (Python Risk Identification Tool for generative AI) is an open-source framework for security professionals to proactively identify risks in generative AI systems.

## Architecture

PyRIT uses a modular "Lego brick" design. The main extensibility points are:

- **Prompt Converters** (`pyrit/prompt_converter/`) — Transform prompts (70+ implementations). Base: `PromptConverter`.
- **Scorers** (`pyrit/score/`) — Evaluate responses. Base: `Scorer`.
- **Prompt Targets** (`pyrit/prompt_target/`) — Send prompts to LLMs/APIs. Base: `PromptTarget`.
- **Executors / Scenarios** (`pyrit/executor/`, `pyrit/scenario/`) — Orchestrate multi-turn attacks.
- **Memory** (`pyrit/memory/`) — `CentralMemory` for prompt/response persistence.

## Code Review Guidelines

When performing a code review, be selective. Only leave comments for issues that genuinely matter:

- Bugs, logic errors, or security concerns
- Unclear code that would benefit from refactoring for readability
- Violations of the critical coding conventions above (async suffix, keyword-only args, type annotations)

Do NOT leave comments about:
- Style nitpicks that ruff/isort would catch automatically
- Missing docstrings or comments — we prefer minimal documentation. Code should be self-explanatory.
- Suggestions to add inline comments, logging, or error handling that isn't clearly needed
- Minor naming preferences or subjective "improvements"

Aim for fewer, higher-signal comments. A review with 2-3 important comments is better than 15 trivial ones.

Follow `.github/instructions/style-guide.instructions.md` for style guidelines. And look in `.github/instructions/` for specific instructions on the different components.
87 changes: 87 additions & 0 deletions .github/instructions/converters.instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
applyTo: "pyrit/prompt_converter/**"
---

# Prompt Converter Development Guidelines

## Base Class Contract

All converters MUST inherit from `PromptConverter` and implement:

```python
class MyConverter(PromptConverter):
SUPPORTED_INPUT_TYPES = ("text",) # Required — non-empty tuple of PromptDataType values
SUPPORTED_OUTPUT_TYPES = ("text",) # Required — non-empty tuple of PromptDataType values

async def convert_async(self, *, prompt: str, input_type: PromptDataType = "text") -> ConverterResult:
...
```

Missing or empty `SUPPORTED_INPUT_TYPES` / `SUPPORTED_OUTPUT_TYPES` raises `TypeError` at class definition time via `__init_subclass__`.

## ConverterResult

Always return a `ConverterResult` with matching `output_type`:

```python
return ConverterResult(output_text=result, output_type="text")
```

Valid `PromptDataType` values: `"text"`, `"image_path"`, `"audio_path"`, `"video_path"`, `"binary_path"`, `"url"`, `"error"`.

The `output_type` MUST match what was actually produced — e.g., if you wrote a file, return the path with `"image_path"`, not `"text"`.

## Input Validation

Check input type support in `convert_async`:

```python
if not self.input_supported(input_type):
raise ValueError(f"Input type {input_type} not supported")
```

## Identifiable Pattern

All converters inherit `Identifiable`. Override `_build_identifier()` to include parameters that affect conversion behavior:

```python
def _build_identifier(self) -> ComponentIdentifier:
return self._create_identifier(
params={"encoding": self._encoding}, # Behavioral params only
children={"target": self._target.get_identifier()} # If converter wraps a target
)
```

Include: encoding types, templates, offsets, model names.
Exclude: retry counts, logging config, timeouts.

## Standard Imports

```python
from pyrit.models import PromptDataType
from pyrit.prompt_converter.prompt_converter import ConverterResult, PromptConverter
from pyrit.identifiers import ComponentIdentifier
```

For LLM-based converters, also import:
```python
from pyrit.prompt_target import PromptChatTarget
```

## Constructor Pattern

Use keyword-only arguments. Use `@apply_defaults` if the converter accepts targets or common config:

```python
from pyrit.common.apply_defaults import apply_defaults

class MyConverter(PromptConverter):
@apply_defaults
def __init__(self, *, target: PromptChatTarget, template: str = "default") -> None:
...
```

## Exports and External Updates

- New converters MUST be added to `pyrit/prompt_converter/__init__.py` — both the import and the `__all__` list.
- The modality table with new/updated converters `doc/code/converters/0_converters.ipynb` and the associated .py pct file must also be updated.
4 changes: 2 additions & 2 deletions .github/instructions/docs.instructions.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
applyTo: 'doc/code/**/*.{py,ipynb}'
applyTo: 'doc/**/*.{py,ipynb}'
---

# Documentation File Synchronization
Expand All @@ -8,7 +8,7 @@ applyTo: 'doc/code/**/*.{py,ipynb}'

All Jupyter notebooks (.ipynb) in the `doc/` directory have corresponding Python (.py) files that are **tightly synchronized**. These files MUST always match exactly in content. They represent the same documentation in different formats.

**Locations:** `doc/code/**/*.ipynb` and `doc/code/**/*.py`
**Locations:** `doc/**/*.ipynb` and `doc/**/*.py`

## Editing Guidelines

Expand Down
Loading