Feat: Content Filtering Exception Handling #3634

AlanPonnachan · 2025-12-03T17:52:16Z

Unified Content Filtering Exception Handling

This PR closes #1035

Standardizing how content filtering events are handled across different model providers.

Previously, triggering a content filter resulted in inconsistent behaviors: generic ModelHTTPError (Azure), UnexpectedModelBehavior (Google), or silent failures depending on the provider. This PR introduces a dedicated exception hierarchy to allow users to catch and handle prompt refusals and response interruptions programmatically and consistently.

Key Changes:

New Exceptions: Added ContentFilterError (base), PromptContentFilterError (for input rejections, e.g., Azure 400), and ResponseContentFilterError (for output refusals).
OpenAI & Azure: Updated logic to raise PromptContentFilterError for Azure's specific 400 error body and ResponseContentFilterError when finish_reason='content_filter'.
Google Gemini: Updated _process_response to raise ResponseContentFilterError instead of UnexpectedModelBehavior when safety thresholds are triggered.
Anthropic: Added mapping for refusal stop reasons to raise ResponseContentFilterError.
Tests: Added comprehensive tests covering synchronous and streaming scenarios for OpenAI, Google, and Anthropic in tests/models/.

Example Usage:

from pydantic_ai import Agent
from pydantic_ai.exceptions import ContentFilterError

agent = Agent('openai:gpt-4o')

try:
    # If the prompt or generation triggers a safety filter
    await agent.run("Generate unsafe content...")
except ContentFilterError as e:
    # Catches both PromptContentFilterError and ResponseContentFilterError
    print(f"Request halted by safety filter: {e}")

rahim-figs · 2025-12-04T12:54:34Z

Is it possible to handle AWS Bedrock as well? Thanks.

dsfaccini

Hey @AlanPonnachan thank you for the PR! I've requested a couple small changes, let me know if you have any questions

pydantic_ai_slim/pydantic_ai/exceptions.py

pydantic_ai_slim/pydantic_ai/models/anthropic.py

pydantic_ai_slim/pydantic_ai/models/openai.py

tests/models/test_anthropic.py

tests/models/test_google.py

tests/models/test_openai.py

dsfaccini · 2025-12-04T19:25:57Z

one more thing I missed, please include the name of the new exception in the fallback model docs

ModelAPIError, which includes ModelHTTPError and ...

AlanPonnachan · 2025-12-05T11:06:46Z

@dsfaccini Thank you for the review. I’ve made the requested changes.

pydantic_ai_slim/pydantic_ai/models/openai.py

tests/models/test_openai.py

pydantic_ai_slim/pydantic_ai/exceptions.py

pydantic_ai_slim/pydantic_ai/models/openai.py

dsfaccini · 2025-12-05T16:31:33Z

hey @AlanPonnachan thanks a lot for your work! It looks very good now, I requested a couple more changes but once that's done and coverage passes I think the PR will be ready for merge.

AlanPonnachan · 2025-12-06T06:33:59Z

@dsfaccini Thanks again for the review! I’ve applied the requested changes. Test coverage is now at 100%.

DouweM

@AlanPonnachan Thanks for working on this! My main concern is that this doesn't actually make it consistent for all models that respond with finish_reason=='content_filter', just for Anthropic/Google/OpenAI.

DouweM · 2025-12-09T19:40:36Z

pydantic_ai_slim/pydantic_ai/exceptions.py

+        self.message = message
+
+
+class PromptContentFilterError(ContentFilterError):


Can we name this RequestContentFilterError for consistency with our ModelRequest/ModelResponse

DouweM · 2025-12-09T19:40:58Z

pydantic_ai_slim/pydantic_ai/exceptions.py

+    """Raised when the prompt triggers a content filter."""
+
+    def __init__(self, status_code: int, model_name: str, body: object | None = None):
+        message = f"Model '{model_name}' content filter was triggered by the user's prompt"


Or by the instructions, right? I'd prefer it to be a bit more generic

DouweM · 2025-12-09T19:46:47Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

            provider_details = {'finish_reason': raw_finish_reason}
            finish_reason = _FINISH_REASON_MAP.get(raw_finish_reason)
+            if finish_reason == 'content_filter':
+                raise ResponseContentFilterError(


I think we should do this in ModelRequestNode, so that it'll work for all models, not just Anthropic.

That would mean that it wouldn't trigger FallbackModel to try another model, but that will be possible once we do #3640.

DouweM · 2025-12-09T19:47:56Z

pydantic_ai_slim/pydantic_ai/models/google.py

            if finish_reason == 'content_filter' and raw_finish_reason:
-                raise UnexpectedModelBehavior(
-                    f'Content filter {raw_finish_reason.value!r} triggered', response.model_dump_json()
+                raise ResponseContentFilterError(


This is a breaking change for users that currently have except UnexpectedModelBehavior. So if we add a new exception, I think it should be a subclass of UnexpectedModelBehavior, not ModelAPIError (as it isn't really an API/HTTP/connection error).

Combined with the above, the solution may be to remove this check here, and to implement the same if response.finish_reason == 'content_filter' check in ModelRequestNode that will apply to all models.

If we want to get some details from the response up to that level, we can store them in response.provider_details

DouweM · 2025-12-09T19:51:01Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+            if (error := body_dict.get('error')) and isinstance(error, dict):
+                error_dict = cast(dict[str, Any], error)
+                if error_dict.get('code') == 'content_filter':
+                    raise PromptContentFilterError(


What do you think about instead handling this by returning a ModelResponse with empty parts and finish_reason == 'content_filter', so that it can go through the same logic as the Anthropic and Google models?

I also don't think they distinguish between input/request and output/response content filter errors, so I don't think we need to here either.

DouweM · 2025-12-11T23:58:08Z

docs/models/overview.md

 By default, the `FallbackModel` only moves on to the next model if the current model raises a
 [`ModelAPIError`][pydantic_ai.exceptions.ModelAPIError], which includes
-[`ModelHTTPError`][pydantic_ai.exceptions.ModelHTTPError]. You can customize this behavior by
+[`ModelHTTPError`][pydantic_ai.exceptions.ModelHTTPError] and [`ContentFilterError`][pydantic_ai.exceptions.ContentFilterError]. You can customize this behavior by


This is no longer true

DouweM · 2025-12-12T00:12:46Z

pydantic_ai_slim/pydantic_ai/_agent_graph.py

        ctx: GraphRunContext[GraphAgentState, GraphAgentDeps[DepsT, NodeRunEndT]],
        response: _messages.ModelResponse,
    ) -> CallToolsNode[DepsT, NodeRunEndT]:
+        if response.finish_reason == 'content_filter':


Before this change, some content filter errors would likely have gone unnoticed because the model could've returned some response parts before the filter was hit and the agent run would've continued as normal, or the model could've returned an empty parts list and we would've auto-retried, which may have succeeded.

So I'm very hesitant to turn this behavior, which is likely working fine for some in production, into a hard error, especially one that can't be recovered by FallbackModel...

So at the very least I think we should only do this if the model returned no parts and it's very unlikely that retrying will help, by moving the check here:

pydantic-ai/pydantic_ai_slim/pydantic_ai/_agent_graph.py

Lines 602 to 609 in eaedf8a

if not self.model_response.parts:

# Don't retry if the model returned an empty response because the token limit was exceeded, possibly during thinking.

if self.model_response.finish_reason == 'length':

model_settings = ctx.deps.model_settings

max_tokens = model_settings.get('max_tokens') if model_settings else None

raise exceptions.UnexpectedModelBehavior(

f'Model token limit ({max_tokens or "provider default"}) exceeded before any response was generated. Increase the `max_tokens` model setting, or simplify the prompt to result in a shorter response that will fit within the limit.'

)

Note that that's also what the GoogleModel logic did.

Of course there are also scenarios where you would want to catch every content filter hit, and always try a different model or fail hard.

The previous ModelAPIError based approach would've worked with FallbackModel, but still feels too aggressive to me (unless we limit it to responses with 0 parts, which would also make it less useful).

So perhaps the upcoming #3640 that will make it very easy to fallback based on response.finish_reason is enough? That in combination with only raising this error if there are 0 parts.

Either way the behavior we end up with will need documentation!

DouweM · 2025-12-12T00:15:39Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+                        chat.chat_completion.Choice(
+                            finish_reason='content_filter',
+                            index=0,
+                            message=chat.ChatCompletionMessage(content='', role='assistant'),


Do we need this? I'd want it to result in a ModelResponse with 0 parts, and this may result in a TextPart(content='')

It may be cleaner to just build that ModelResponse directly...

AlanPonnachan and others added 2 commits December 3, 2025 17:35

add content filtering exceptions

4ac81d4

Merge branch 'main' into feature/content-filtering-exceptions

658f407

dsfaccini requested changes Dec 4, 2025

View reviewed changes

AlanPonnachan and others added 5 commits December 5, 2025 09:31

resolve issues

404a833

add docs

3771c79

Merge branch 'main' into feature/content-filtering-exceptions

28e20c8

test coverage

70bcb74

improve tests

e861a50

dsfaccini requested changes Dec 5, 2025

View reviewed changes

AlanPonnachan added 3 commits December 6, 2025 05:13

resolve issues

51506b2

test coverage

b67d216

fix ci fails

4ac1608

DouweM requested changes Dec 9, 2025

View reviewed changes

DouweM self-assigned this Dec 9, 2025

DouweM added the awaiting author revision label Dec 9, 2025

AlanPonnachan and others added 2 commits December 11, 2025 17:30

make consistent for all models

264f9e6

Merge branch 'main' into feature/content-filtering-exceptions

ed242de

DouweM requested changes Dec 12, 2025

View reviewed changes

		self.message = message


		class PromptContentFilterError(ContentFilterError):

	if not self.model_response.parts:
	# Don't retry if the model returned an empty response because the token limit was exceeded, possibly during thinking.
	if self.model_response.finish_reason == 'length':
	model_settings = ctx.deps.model_settings
	max_tokens = model_settings.get('max_tokens') if model_settings else None
	raise exceptions.UnexpectedModelBehavior(
	f'Model token limit ({max_tokens or "provider default"}) exceeded before any response was generated. Increase the `max_tokens` model setting, or simplify the prompt to result in a shorter response that will fit within the limit.'
	)

Feat: Content Filtering Exception Handling #3634

Are you sure you want to change the base?

Feat: Content Filtering Exception Handling #3634

Conversation

AlanPonnachan commented Dec 3, 2025

Unified Content Filtering Exception Handling

Uh oh!

rahim-figs commented Dec 4, 2025

Uh oh!

dsfaccini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsfaccini commented Dec 4, 2025

Uh oh!

AlanPonnachan commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsfaccini commented Dec 5, 2025

Uh oh!

AlanPonnachan commented Dec 6, 2025

Uh oh!

DouweM left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants