-
Notifications
You must be signed in to change notification settings - Fork 0
Add multi-provider support (OpenAI, Gemini, Mistral/OpenRouter) #1
Copy link
Copy link
Open
Labels
good first issueGood for newcomersGood for newcomers
Description
Summary
Currently the benchmark runs against a single LLM provider. Add support for running the same benchmark against OpenAI (GPT-4o), Google Gemini (2.5 Pro), and open-source models via Mistral/OpenRouter to enable cross-provider comparison.
What needs to happen
- Add provider configuration (env vars or config file) for OpenAI, Gemini, and Mistral/OpenRouter
- Abstract the LLM call layer so provider can be swapped without changing benchmark logic
- Add a
--providerflag or config option to select which provider to run against - Run the benchmark on each provider and record results
- Document required API keys and setup for each provider in README
Acceptance criteria
- Benchmark can run against at least 3 providers (Claude, OpenAI, Gemini)
- Results are recorded in a comparable format regardless of provider
- README documents how to configure and run with each provider
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
good first issueGood for newcomersGood for newcomers