Create LiteLLMModel to fix thinking parts not being sent to Anthropic on Vertex via LiteLLM with OpenAIChatModel

[ExtraBB]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

Hi,

I'm using a LiteLLM proxy which proxies requests for claude models to Google Vertex AI. However, I am hitting errors when toolcalls and thinking are getting in the mix:

ERROR:src.agents.excel.excel_agent_message_processor:Exception cause: BadRequestError: Error code: 400 - {'error': {'message': 'litellm.BadRequestError: Vertex_aiException BadRequestError - b\'{"type":"error","error":{"type":"invalid_request_error","message":"messages.1.content.0.type: Expected `thinking` or `redacted_thinking`, but found `text`. When `thinking` is enabled, a final `assistant` message must start with a thinking block (preceeding the lastmost set of `tool_use` and `tool_result` blocks). We recommend you include thinking blocks from previous turns. To avoid this requirement, disable `thinking`. Please consult our documentation at https://docs.claude.com/en/docs/build-with-claude/extended-thinking"},"request_id":"req_vrtx_011CTupmw9UhTioFUQDwomKa"}\'. Received Model Group=anthropic/claude\nAvailable Model Group Fallbacks=None', 'type': None, 'param': None, 'code': '400'}}

This looks similar to the issues:

• Send ThinkingPart back for OpenAIChatModel models which require reasoning content #2701
• Anthropic extended thinking: invalid request error when using tools #2040

I am at a bit of a loss what I can do here, any ideas/suggestions? I would expect pydantic to send back the thinking blocks when there are thinkingparts. I also added an example of openai working normally with the same endpoint.

PS: @DouweM Long time no see! We worked together on Convex at Utrecht University in another lifetime :D

Example Code

Not working :

#!/usr/bin/env python3
"""
Standalone test script for litellm server with Pydantic AI.
Tests extended thinking and tool calling.
"""

from pydantic_ai import Agent, ModelSettings, RunContext
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.litellm import LiteLLMProvider
from pydantic import BaseModel
import pprint


# Define weather data structure
class WeatherData(BaseModel):
    temperature: float
    conditions: str
    unit: str


# Configure the model
model = OpenAIChatModel(
    "anthropic/claude-sonnet-4-5-20250929",
    provider=LiteLLMProvider(
        api_base="<endpoint>",
        api_key="<key>",
    ),
    settings=ModelSettings(
        max_tokens=4096,
        extra_body={
            "thinking": {
                "type": "enabled",
                "budget_tokens": 1024
            }
        }
    )
)

# Create the agent
agent = Agent(
    model,
    system_prompt="You are a helpful weather assistant. Use the get_weather tool to retrieve weather information."
)


# Define the tool
@agent.tool
def get_weather(ctx: RunContext, location: str, unit: str = "celsius") -> WeatherData:
    """
    Get the current weather for a given location.

    Args:
        location: The city and state, e.g. San Francisco, CA
        unit: The temperature unit to use (celsius or fahrenheit)
    """
    print(f"\n--- Tool Called: get_weather ---")
    print(f"Location: {location}")
    print(f"Unit: {unit}")

    # Return dummy weather data
    return WeatherData(
        temperature=18,
        conditions="partly cloudy",
        unit=unit
    )


async def main():
    try:
        # Run the agent
        result = await agent.run(
            "What's the weather like in Paris? Think through your answer carefully."
        )

        print("\n--- Messages ---")
        pprint.pp(result.all_messages())

        print("\n Test completed successfully!")

    except Exception as e:
        print(f"\n Error occurred: {type(e).__name__}")
        print(f"Message: {str(e)}")
        raise


if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Working with pure openai

#!/usr/bin/env python3
"""
Standalone test script for litellm server with OpenAI SDK.
Tests extended thinking and tool calling.
"""

from openai import OpenAI
import pprint

# Configure client for litellm server
client = OpenAI(
    base_url="<endpoint>",
    api_key="<key>",
)

# Define a test tool
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use",
                    },
                },
                "required": ["location"],
            },
        },
    }
]

def main():
    print("Testing litellm server with OpenAI SDK...")

    try:
        # Initialize messages
        messages = [
            {
                "role": "user",
                "content": "What's the weather like in Paris? Think through your answer carefully.",
            }
        ]

        # Make a request with extended thinking and tool calling
        response = client.chat.completions.create(
            model="anthropic/claude-sonnet-4-5-20250929",
            messages=messages,
            tools=tools,
            # Enable extended thinking for Claude
            extra_body={
                "thinking": {
                    "type": "enabled",
                    "budget_tokens": 1024
                }
            }
        )

        pprint.pp(response.choices[0].message)
        # Check for thinking content
        message = response.choices[0].message
        if hasattr(message, 'content') and message.content:
            print(f"\n--- Content ---")
            print(message.content)

        # Add assistant's response to messages
        messages.append(message.model_dump(exclude_unset=True))

        # Check for tool calls
        if hasattr(message, 'tool_calls') and message.tool_calls:
            print(f"\n--- Tool Calls ---")
            for tool_call in message.tool_calls:
                print(f"Tool: {tool_call.function.name}")
                print(f"Arguments: {tool_call.function.arguments}")

            # Respond to tool call with dummy value
            print("\n--- Responding to tool call with dummy data ---")

            # Add tool result to messages
            messages.append({
                "role": "tool",
                "tool_call_id": message.tool_calls[0].id,
                "content": '{"temperature": 18, "conditions": "partly cloudy", "unit": "celsius"}'
            })

            print("SENDING MESSAGES:")
            pprint.pp(messages)

            # Get final answer
            final_response = client.chat.completions.create(
                model="anthropic/claude",
                messages=messages,
                tools=tools,
                extra_body={
                    "thinking": {
                        "type": "enabled",
                        "budget_tokens": 1024
                    }
                }
            )

            print("\n--- Final Answer ---")
            pprint.pp(final_response.choices[0].message)
            final_message = final_response.choices[0].message
            if hasattr(final_message, 'content') and final_message.content:
                print(final_message.content)

        print("\n Test completed successfully!")

    except Exception as e:
        print(f"\n Error occurred: {type(e).__name__}")
        print(f"Message: {str(e)}")
        raise

if __name__ == "__main__":
    main()

Python, Pydantic AI & LLM client version

Python 3.12.7
pydantic-ai 1.0.15

[DouweM]

@ExtraBB Hey Bruno, gaaf je hier te zien! :D

Pydantic AI -> OpenAI Chat Completions API -> LiteLLM -> Vertex AI -> Claude forces the data through a couple of leaky abstractions.

Would it be an option to use GoogleModel (docs) or AnthropicModel (with AsyncAnthropicVertex as anthropic_client, see #960 (comment)), so Pydantic AI ends up talking to the Vertex AI API directly?

Both those model classes have native types for sending and receiving reasoning parts; the OpenAI Chat Completions API and types from the SDK don't, so we have to do things like this to read the reasoning anyway:

pydantic-ai/pydantic_ai_slim/pydantic_ai/models/openai.py

Lines 544 to 557 in 3eaa11e

	# The `reasoning_content` field is only present in DeepSeek models.
	# https://api-docs.deepseek.com/guides/reasoning_model
	if reasoning_content := getattr(choice.message, 'reasoning_content', None):
	items.append(ThinkingPart(id='reasoning_content', content=reasoning_content, provider_name=self.system))
	# The `reasoning` field is only present in gpt-oss via Ollama and OpenRouter.
	# - https://cookbook.openai.com/articles/gpt-oss/handle-raw-cot#chat-completions-api
	# - https://openrouter.ai/docs/use-cases/reasoning-tokens#basic-usage-with-reasoning-tokens
	if reasoning := getattr(choice.message, 'reasoning', None):
	items.append(ThinkingPart(id='reasoning', content=reasoning, provider_name=self.system))
	# NOTE: We don't currently handle OpenRouter `reasoning_details`:
	# - https://openrouter.ai/docs/use-cases/reasoning-tokens#preserving-reasoning-blocks
	# If you need this, please file an issue.

Note that these are not fields that really exist on the types, so we have to use getattr. Similarly, the fields don't exist on the types we send to the API, so it'd be risky to setattr them as some Chat Completions-"compatible" APIs like LiteLLM may accept them, but others may raise an error or want them in another (also technically incompatible with the OpenAI spec) format.

From https://docs.litellm.ai/docs/reasoning_content#spec, it sounds like LiteLLM sort of works because of reasoning_content, but we should be reading thinking_blocks as well. It's not explicitly said which field we should be setting on the API request body to prevent the "When thinking is enabled, a final assistant message must start with a thinking block" error you're seeing, but as messages.append(message.model_dump(exclude_unset=True)) worked in your openai-only example, that's probably thinking_blocks (or both).

We could add some configuration to LiteLLMProvider to let the OpenAIChatModel know that it supports reading and returning thinking_parts, but as this really has little to do with OpenAI anymore, it maybe time for a LiteLLMModel subclass of OpenAIChatModel, similar to the upcoming OpenRouterModel: #2936. (OpenRouter, of course, implements its own custom reasoning fields on top of the Chat Completions API spec.)

You'll save yourself a good amount of headaches using APIs and SDKs that officially support all the features you need, instead of them being somewhat hackily bolted on top :) But I understand there are scenarios where you have to use LiteLLM. If using one of the other APIs is not an option, let me know and I can guide you on creating a PR to fix this, if you're interested.

[ExtraBB]

Hey thanks for the quick answer! We will try to see if we can use LiteLLM's passthrough endpoint so we can use AnthropicVertex directly. I do think the LiteLLMModel subclass makes sense. Google ADK has first class LiteLLM support for example. I'd be interested in contributing if you can give me some pointers!

[DouweM]

@ExtraBB Let me know how it goes with the pass-through endpoint. Then for LiteLLMModel, I suggest waiting for OpenRouterModel in #3089 as it'll provide a nice blueprint.

[ExtraBB]

@DouweM The Passthrough endpoints work as expected. I prefer this over an extra unnecessary OpenAI abstraction layer so I don't think a LiteLLM model is needed for us. Do you know of other use cases warranting a specific LiteLLM model?

[DouweM]

@ExtraBB I haven't seen any yet, but I'm sure that like OpenRouter they support some features we can't currently support in OpenAIChatModel so we can leave this issue open for when someone else runs into an issue and can't move off of LiteLLM.