Add Cloudflare AI Gateway provider #3199
Conversation
Implements CloudflareProvider for routing requests through Cloudflare's unified AI Gateway API with support for: - Multiple AI providers (OpenAI, Anthropic, Groq, Mistral, Cohere, etc.) - BYOK (bring your own key) mode - Stored keys mode (API keys managed in Cloudflare dashboard) - Authenticated gateways with cf-aig-authorization header - Intelligent model profiling for Groq and Cerebras models Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com>
Add missing import statements to examples 2 and 3 in the class docstring to resolve Ruff F821 errors (undefined name CloudflareProvider). Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com>
Replace broken Agent('model', provider=provider) pattern with the correct
OpenAIChatModel pattern in all three usage examples. This matches the fix
already applied to docs/models/openai.md.
Generated with Claude Code
https://claude.com/claude-code
Co-Authored-By: Claude <noreply@anthropic.com>
|
The CI checks are passing now, lmk what you think and if there's anything you'd like to change. I'm attaching my notebook where I tested with my gateway. test-cloudflare-provider.html. A couple irregularities there: since I'm instatiating the Groq and Cerebras providers to get the right profile they naturally look for their respective api keys in the env. Or at least Groq does, idk why Cerebras doesn't. Second irregularity: the google genai api returns a 400 on missing auth (???) Have a great week! |
| Cloudflare routes to Groq's OpenAI-compatible endpoint, so we use prefix matching | ||
| similar to the native GroqProvider to determine the appropriate profile. | ||
| """ | ||
| return GroqProvider().model_profile(model_name) |
There was a problem hiding this comment.
Let's move this into a groq_model_profile function defined in groq.py so we don't have to instantiate the provider class.
There was a problem hiding this comment.
I was wondering if we could set the GroqProvider::model_profile as a @statichmethod? It doesn't use self at all. That way we don't have to instatiate it. See here
There was a problem hiding this comment.
or should I move the logic completely from providers/groq.py::GroqProvider.model_profile to profiles/groq.py::groq_model_profile? I can open a smaller PR for that. Would add the cerebras and perplexity profile with it as well.
There was a problem hiding this comment.
Under profiles, we should only have files for specific model families. Groq and Cerebras are providers, and Cloudflare is so-far unique among providers/gateways in that it routes to other gateways (I'm understanding that correctly right?), so having them in the providers module makes sense. I'd prefer top-level functions to copy the pattern set by the profiles modules. Feel free to do this in this PR.
There was a problem hiding this comment.
I'll do what you're asking:
- extract the
providers/groq.py::GroqProvider.model_profile - into a top level function
providers/groq.py::model_profile
One thing I noted: providers/litellm.py imports profiles/groq.py#L19, and the respective test uses the one imported by the provider (the same profiles/groq.py). So it seems to me they're both using the wrong groq_model_profile function.
I wrote a test for this here https://gist.github.com/dsfaccini/23ad7695043329d81bef59907ba26aaf
Show code
# pydantic_ai_slim/pydantic_ai/providers/litellm.py#L16
from pydantic_ai.profiles.groq import groq_model_profile
# pydantic_ai_slim/pydantic_ai/profiles/groq.py#L19
def groq_model_profile(model_name: str) -> ModelProfile:
"""Get the model profile for a Groq model."""
return GroqModelProfile(
groq_always_has_web_search_builtin_tool=model_name.startswith('compound-'),
)Should I update that import too?
There was a problem hiding this comment.
Ah so we have a groq_model_profile already... Well in that case, just create new files under profiles for all the ones we're missing
There was a problem hiding this comment.
I left this as is for now, just extracted the method from inside the class into a top-level function. I thought this would be easier for you to review.
If this works for you I'll go ahead and move them into the own profiles/*.py files and update the two imports by the litellm modules.
This commit implements all 9 review comments from @DouweM: 1. Add cloudflare: model name shorthand support - Added 'cloudflare' to OpenAI-compatible providers in models/__init__.py - Added 10 representative Cloudflare model names to KnownModelName - Added shorthand usage example to docs 2-3. Extract Groq/Cerebras model profiling to top-level functions - Created groq_provider_model_profile() in providers/groq.py - Created cerebras_provider_model_profile() in providers/cerebras.py - Updated CloudflareProvider to use direct imports instead of instantiation 4. Add perplexity_model_profile function - Created profiles/perplexity.py with perplexity_model_profile() - Updated CloudflareProvider to use it 5. Remove AI-generated comment from cloudflare.py 6. Fix HTTP client monkeypatching - Use empty string for api_key in CF-managed keys mode - Removed _create_stored_keys_client() method (~26 lines) - Leverages OpenAI SDK's built-in behavior 7. Rename cf_aig_authorization → gateway_auth_token - Updated parameter name throughout codebase - HTTP header stays 'cf-aig-authorization' (Cloudflare API requirement) 8. Remove use_gateway_keys parameter - Made CF-managed keys mode detection implicit - Updated overload signatures - Simplified initialization logic - Deleted 2 obsolete tests Also updated terminology from "BYOK/stored keys" to "user-managed/CF-managed" for clarity and to avoid confusion with Cloudflare's BYOK feature. All tests passing (16 Cloudflare tests, 100% coverage maintained). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
|
|
||
| # Set via environment or in code: | ||
| # CLOUDFLARE_ACCOUNT_ID='your-account-id' | ||
| # CLOUDFLARE_GATEWAY_ID='your-gateway-id' | ||
| # OPENAI_API_KEY='your-openai-api-key' |
There was a problem hiding this comment.
| # Set via environment or in code: | |
| # CLOUDFLARE_ACCOUNT_ID='your-account-id' | |
| # CLOUDFLARE_GATEWAY_ID='your-gateway-id' | |
| # OPENAI_API_KEY='your-openai-api-key' |
| 'cloudflare:google/gemini-2.0-flash', | ||
| 'cloudflare:groq/llama-3.3-70b-versatile', | ||
| 'cloudflare:mistral/mistral-large-latest', | ||
| 'cloudflare:openai/gpt-4o', |
There was a problem hiding this comment.
Why are more recent models missing?
| openai_names = [f'openai:{n}' for n in get_model_names(OpenAIModelName)] | ||
| bedrock_names = [f'bedrock:{n}' for n in get_model_names(BedrockModelName)] | ||
| deepseek_names = ['deepseek:deepseek-chat', 'deepseek:deepseek-reasoner'] | ||
| cloudflare_names = [ |
There was a problem hiding this comment.
As mentioned above, more modern names are missing. I'd rather NOT include this list here if we can't get it dynamically from the API or SDK.
2 participants
Adds support for Cloudflare AI Gateway as a provider.
Hey guys, long time lurker first time committer, I read this issue #1381 and saw it hadn't moved since April, so I gave it a shot. Having used the CF AIG myself I thought I'd be able to debug any issues.
I did use Claude Code to add this feature, but I tried to stick to the syntax of similar implementations (Vercel Gateway, OpenRouter), reviewed the code manually, and tested this branch locally with my own gateway. The test file is a bit longer than the rest and
cloudflare.pyhas more docstrings and comments, but I thought this to be appropriate, as this provider is more complex given the usage of stored keys.I do see that there are CI checks that are failing so I'll address those and re-commit
I'll leave the description + some good to knows below:
Features
BYOK)Implementation notes
Follows the established pattern from
VercelProviderandOpenRouterProvider. The stored keys feature is unique to Cloudflare and required additional logic to strip the Authorization header when using keys stored in the Cloudflare dashboard.Testing