Skip to content
This repository was archived by the owner on Mar 14, 2026. It is now read-only.

feat: Add LiteLLM Proxy Integration for Multi-Provider LLM Support#34

Merged
takasaki404 merged 26 commits intomainfrom
wesley-working-branch
Sep 6, 2025
Merged

feat: Add LiteLLM Proxy Integration for Multi-Provider LLM Support#34
takasaki404 merged 26 commits intomainfrom
wesley-working-branch

Conversation

@Wes1eyyy
Copy link
Contributor

@Wes1eyyy Wes1eyyy commented Aug 21, 2025 •
edited
Loading

Title: feat: Add LiteLLM Proxy Integration for Multi-Provider LLM Support

Description:

Summary

Adds LiteLLM Proxy integration to enable routing to multiple LLM providers through a unified API interface while maintaining full backward compatibility.

Changes

  • New LLM Client (app/clients/llm_client.py) - Async client for LiteLLM Proxy communication
  • New API Routes - /chat/completions_proxy, /chat/models, /chat/health endpoints
  • Configuration - Environment variables for LiteLLM settings in app/core/config.py
  • Infrastructure - Docker Compose setup for LiteLLM Proxy (infra/litellm/)
  • Generic Models - Provider-agnostic data models (app/models/llm.py)
  • Enhanced Error Handling - LLM-specific error codes (401, 429, 502, 504)

Key Features

  • 🔄 Multi-provider support (OpenAI, Anthropic, OpenRouter, Ollama)
  • 🛡️ Automatic fallback between providers
  • 💰 Cost optimization through smart routing
  • 🏠 Local model integration support
  • 📊 Built-in monitoring and analytics

Backward Compatibility

  • ✅ All existing routes unchanged
  • ✅ Original /chat/completions endpoint preserved
  • ✅ app/ai/aimo.py remains untouched
  • ✅ No breaking changes to existing functionality

Setup

# Start LiteLLM Proxy
cd infra/litellm
docker-compose -f docker-compose.litellm.yml up -d

# Verify integration
curl http://localhost:8000/api/v1.0.0/chat/health

Wes1eyyy added 26 commits August 20, 2025 13:28
@Wes1eyyy
refactor(db): improve database URL handling for testing environment
356172b
@Wes1eyyy
feat(llm): implement LLMClient for asynchronous chat interactions and…
903b75c 
… add configuration for models
@Wes1eyyy
feat: Implement LiteLLM Proxy integration for chat completions
e914d54 
- Added new endpoint `/completions_proxy` to handle chat completions via LiteLLM Proxy.
- Introduced LLMClient for unified access to multiple LLM providers.
- Implemented streaming and non-streaming responses for chat completions.
- Added endpoints for listing available models and health checks.
- Created standardized data models for LLM requests and responses.
- Updated configuration settings for LiteLLM Proxy.
- Added exception handling for OpenAI-specific errors.
- Implemented tests for the new chat proxy functionality and LLM client.
- Ensured backward compatibility with existing OpenAI models.
@Wes1eyyy
feat: Refactor LiteLLM configuration and remove deprecated client code
5f4731d
@Wes1eyyy
feat: Update LiteLLM configuration for improved model handling and en…
2878877 
…vironment variable usage
@Wes1eyyy
feat: Add LiteLLM environment configuration and update docker-compose…
ba0d642 
… to use it
@Wes1eyyy
feat: Add LiteLLM Proxy integration documentation and setup instructi…
b705e47 
…ons in README
@Wes1eyyy
feat: Enhance LiteLLM configuration with additional models and fallba…
568d103 
…ck settings
@Wes1eyyy
feat: Update docker-compose configuration for improved volume managem…
b7cc0d9 
…ent and formatting
@Wes1eyyy
feat: Add .gitkeep file to ensure directory structure and update Lite…
a913e8d 
…LLM config for database logging
@Wes1eyyy
feat: Refactor docker-compose configuration for improved readability …
3bf493f 
…and structure
@Wes1eyyy
feat: Enable admin API in general settings for enhanced management ca…
c2953c6 
…pabilities
@Wes1eyyy
feat: Update general settings in LiteLLM config to enable database lo…
0cd4f4e 
…gging and specify database URL
@Wes1eyyy
feat: Move enable_admin_api setting to router_settings for better org…
0e49174 
…anization
@Wes1eyyy
fix: Correct model version for deepseek_r1_052 in LiteLLM config
aba0735
@Wes1eyyy
feat: Add manage-virtual-keys option to litellm-proxy command for enh…
40e6a7f 
…anced functionality
@Wes1eyyy
fix: Standardize model names in litellm_config.yaml for consistency
04b6eb5
@Wes1eyyy
fix: Correct model name format for openai/gpt-oss-20b in litellm_conf…
2e737dc 
…ig.yaml
@Wes1eyyy
fix: Correct model name format for openai/gpt-oss-20b-free in litellm…
8aa4cbc 
…_config.yaml
@Wes1eyyy
fix: Remove manage-virtual-keys option from litellm-proxy command in …
846d0bf 
…docker-compose
@Wes1eyyy
fix: Update port configuration for litellm-proxy service in docker-co…
820ff26 
…mpose
@Wes1eyyy
fix: Update port configuration for litellm-proxy service to use port …
6b60b32 
…4000
@Wes1eyyy
fix: Enable verbose logging in litellm_settings and add logging confi…
45c9118 
…guration
@Wes1eyyy
fix: Remove version declaration from docker-compose and add fallbacks…
4f35e7b 
… to model configurations in litellm_config.yaml
@Wes1eyyy
fix: Update model names and parameters in litellm_config.yaml for dee…
769c920 
…pseek-r1 and glm-4_5
@Wes1eyyy
fix: Add qwen3-235b-a22b model configuration with fallbacks in litell…
51bb8cb 
…m_config.yaml
@takasaki404 takasaki404 merged commit ca250f9 into main Sep 6, 2025
1 check failed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@takasaki404 takasaki404 takasaki404 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Wes1eyyy @takasaki404