Skip to content

Conversation

habema
Copy link
Contributor

@habema habema commented Sep 30, 2025

Resolves #1831

Adds cost tracking for LiteLLM models when users enable it via ModelSettings(track_cost=True).

Changes

  • Added cost field to Usage class
  • Added track_cost setting to ModelSettings. Opt-in (defaults to False)
  • LiteLLM cost extraction:
    • Non-streaming: reads from _hidden_params["response_cost"]
    • Streaming: calculates via litellm.completion_cost()
    • Includes cached & reasoning tokens for accurate pricing

@seratch seratch added enhancement New feature or request feature:lite-llm labels Sep 30, 2025
@seratch seratch marked this pull request as draft September 30, 2025 18:12
Copy link
Member

@seratch seratch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlocking the access itself is fine, but we'd like to hold off relying on an internal modules within a different package.

# Extract cost from LiteLLM's hidden params if cost tracking is enabled.
cost = None
if model_settings.track_cost:
if hasattr(response, "_hidden_params") and isinstance(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd like to avoid accessing the internal modules

@seratch seratch changed the title Add opt-in cost tracking for LiteLLM models feat: #1831 Add opt-in cost tracking for LiteLLM models Sep 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request feature:lite-llm
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add cost tracking for LiteLLM models
2 participants