Add `force_download` support for Anthropic and OAIR models and clarify proper `BinaryContent` base64 handling #3694

dsfaccini · 2025-12-10T12:50:25Z

Closes #3569 - Review multi-modal input handling.

Changes

BinaryContent improvements:

Add base64 property that returns raw base64-encoded string
Update data field docstring to clarify it should contain raw bytes, not base64-encoded strings

Anthropic provider:

Add force_download support for ImageUrl - downloads and sends as base64 when force_download=True
Add text/plain support for BinaryContent - decodes UTF-8 and sends as plain text document
Extract _map_binary_content() helper to reduce complexity of _map_user_prompt()

Test fixes:

Fix test_mcp_sampling.py: was storing base64.b64encode(b'data') instead of raw b'data'
Fix test_mistral.py: was storing base64 ASCII characters as bytes
Update test_anthropic.py error message match for audio content

OpenAI Responses API:

Add force_download support for AudioUrl and DocumentUrl - uses file_url to send URLs directly when force_download=False (default), downloads and sends as file_data when force_download=True
Consolidate AudioUrl and DocumentUrl handling into single branch

New tests:

test_binary_content_base64 - verifies new property
test_image_url_input_force_download - verifies Anthropic force_download
test_text_document_as_binary_content_input - verifies Anthropic text/plain support
test_document_url_input_response_api - verifies URL is sent directly (default behavior)
test_document_url_input_force_download_response_api - verifies download when force_download=True

github-actions · 2025-12-10T13:06:56Z

Docs Preview

commit:	`35d8745`
Preview URL:	https://6895a4a3-pydantic-ai-previews.pydantic.workers.dev

dsfaccini · 2025-12-10T14:44:27Z

work's still ongoing to address the other points mentioned in the issue

pydantic_ai_slim/pydantic_ai/models/anthropic.py

DouweM · 2025-12-10T20:46:37Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+                    yield AnthropicModel._map_binary_content(item)
+                elif isinstance(item, ImageUrl):
+                    if item.force_download:
+                        downloaded = await download_item(item, data_format='bytes')


We should also respect force_download for DocumentUrl + item.media_type == 'application/pdf' further down, right?

Would it make sense to use _map_binary_content there, if we make it more generic so it can take the result of download_item? (Or take a FileUrl | BinaryContent and if it gets a FileUrl, do the download first)

DouweM · 2025-12-10T20:47:38Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+                type='image',
+            )
+        elif item.media_type == 'application/pdf':
+            return BetaBase64PDFBlockParam(


This is now an alias for BetaRequestDocumentBlockParam, so let's use the newer name

DouweM · 2025-12-10T20:49:49Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+            )
+        elif item.media_type == 'application/pdf':
+            return BetaBase64PDFBlockParam(
+                source=BetaBase64PDFSourceParam(


I'm looking at what other sources this supports, and there's also BetaFileDocumentSourceParam, which takes a file_id for the file upload API.

We're adding support for uploaded files in #2611, but that PR has been stale for a bit so may be interesting for you to pick up.

DouweM · 2025-12-10T20:53:46Z

tests/models/test_anthropic.py

+            ),
+        ]
+    )
+    assert result.output == snapshot(


Can we test that the forced download is actually happening?

DouweM · 2025-12-10T20:56:36Z

tests/models/test_openai.py

+
+    document_url = DocumentUrl(
+        url='https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf',
+        force_download=True,


Same as up -- we're not testing that we're respecting this, although I suppose that fact that those lines have test coverage sort of does? But it'd be good if we could assert that explicitly

pydantic_ai_slim/pydantic_ai/messages.py

This commit addresses Douwe's review comments on PR #3694: 1. **Anthropic: Add force_download for DocumentUrl PDF** - DocumentUrl with PDF now respects force_download parameter - When force_download=True: downloads and sends as base64 - When force_download=False: sends URL directly 2. **Anthropic: Rename BetaBase64PDFBlockParam → BetaRequestDocumentBlockParam** - Updated to use newer type name (BetaBase64PDFBlockParam is now an alias) - Updated imports and all usages 3. **Anthropic: Refactor _map_binary_content to handle FileUrl** - Created new _map_binary_to_block helper method - Eliminates code duplication between BinaryContent, ImageUrl force_download, and DocumentUrl force_download - Supports image/*, application/pdf, and text/plain media types 4. **Messages: Use base64 property in otel_message_parts** - Replaced manual base64.b64encode calls with item.base64 property - 3 replacements in UserPromptPart.otel_message_parts and ModelResponse.otel_* methods 5. **Tests: Add force_download verification tests** - 4 new tests for Anthropic (ImageUrl + DocumentUrl) - 5 new tests for OpenAI (ImageUrl + DocumentUrl + AudioUrl) - Tests verify download_item is called when force_download=True All tests pass (306 tests in test_anthropic.py, test_openai.py, test_messages.py). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

DouweM · 2025-12-12T20:42:52Z

pydantic_ai_slim/pydantic_ai/messages.py


    data: bytes
-    """The binary data."""
+    """Arbitrary binary data.


"Arbitrary" may be a little too strong, as it is specifically a file matching the type in media_type 😄

I think if we make this The binary file data, we can remove the specific mention of base64 that's added in the next lines

I'd like to keep one as it reduces the likelihood of users using it incorrectly

DouweM · 2025-12-12T20:44:37Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+            raise RuntimeError(f'Unsupported binary content media type for Anthropic: {media_type}')
+
+    @staticmethod
+    def _map_binary_content(item: BinaryContent) -> BetaContentBlockParam:


We don't need this method anymore as it's just one line that can be inlined

I wonder if download_item could be refactored to return a BinaryContent

would've been cool if it had been like BinaryContent.download or .from_url but I don't think the replacement is worth the hassle now

pydantic_ai_slim/pydantic_ai/models/anthropic.py

dsfaccini · 2025-12-13T04:26:27Z

changed the docs as follows https://29ed66a3-pydantic-ai-previews.pydantic.workers.dev/input

force_download snippet -> better than describing it (personally, but also for llms)

review multi modal input handling

c642ac7

dsfaccini marked this pull request as draft December 10, 2025 14:44

enable url media for openai responses

754c0f0

dsfaccini changed the title ~~Add force_download support for anthropic model and clarify proper BinaryContent base64 handling~~ Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling Dec 10, 2025

Update documentation for BinaryContent.base64 usage

2f5bcf6

dsfaccini marked this pull request as ready for review December 10, 2025 18:02

dsfaccini changed the title ~~Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling~~ Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling Dec 10, 2025

dsfaccini marked this pull request as draft December 10, 2025 18:06

dsfaccini commented Dec 10, 2025

View reviewed changes

pydantic_ai_slim/pydantic_ai/models/anthropic.py Show resolved Hide resolved

dsfaccini marked this pull request as ready for review December 10, 2025 18:21

dsfaccini requested a review from DouweM December 10, 2025 18:22

dsfaccini added 2 commits December 10, 2025 13:22

Merge branch 'main' into review-multimodal

5d99ed2

Merge branch 'main' into review-multimodal

ff5d79a

DouweM requested changes Dec 10, 2025

View reviewed changes

DouweM self-assigned this Dec 10, 2025

DouweM added the awaiting author revision label Dec 10, 2025

dsfaccini and others added 5 commits December 10, 2025 17:48

Merge branch 'main' into review-multimodal

887a79f

upstream cerebras merge stuff

7389467

AsyncClient

8ad810d

base64 replacements

da9ec78

dsfaccini mentioned this pull request Dec 11, 2025

Cohere: Implement multimodal support for Command A Vision #3703

Open

add force download support for mistral

9a55ce2

DouweM requested changes Dec 12, 2025

View reviewed changes

dsfaccini added 2 commits December 12, 2025 19:58

address review comments

703b772

update docs

35d8745

Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling #3694

Are you sure you want to change the base?

Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling #3694

Conversation

dsfaccini commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docs Preview

Uh oh!

dsfaccini commented Dec 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dsfaccini commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `force_download` support for Anthropic and OAIR models and clarify proper `BinaryContent` base64 handling #3694

Add `force_download` support for Anthropic and OAIR models and clarify proper `BinaryContent` base64 handling #3694

dsfaccini commented Dec 10, 2025 •

edited

Loading

github-actions bot commented Dec 10, 2025 •

edited

Loading

dsfaccini commented Dec 13, 2025 •

edited

Loading