Skip to content

Add multi-provider support (OpenAI, Gemini, Mistral/OpenRouter) #1

@rajkumar42

Description

@rajkumar42

Summary

Currently the benchmark runs against a single LLM provider. Add support for running the same benchmark against OpenAI (GPT-4o), Google Gemini (2.5 Pro), and open-source models via Mistral/OpenRouter to enable cross-provider comparison.

What needs to happen

  • Add provider configuration (env vars or config file) for OpenAI, Gemini, and Mistral/OpenRouter
  • Abstract the LLM call layer so provider can be swapped without changing benchmark logic
  • Add a --provider flag or config option to select which provider to run against
  • Run the benchmark on each provider and record results
  • Document required API keys and setup for each provider in README

Acceptance criteria

  • Benchmark can run against at least 3 providers (Claude, OpenAI, Gemini)
  • Results are recorded in a comparable format regardless of provider
  • README documents how to configure and run with each provider

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions