Skip to content

fix: use max_tokens for openai provider with custom base URL (Mistral compatibility)#858

Open
octo-patch wants to merge 1 commit intovectorize-io:mainfrom
octo-patch:fix/issue-852-use-max-tokens-for-custom-base-url
Open

fix: use max_tokens for openai provider with custom base URL (Mistral compatibility)#858
octo-patch wants to merge 1 commit intovectorize-io:mainfrom
octo-patch:fix/issue-852-use-max-tokens-for-custom-base-url

Conversation

@octo-patch
Copy link
Copy Markdown
Contributor

Fixes #852

Problem

When using HINDSIGHT_API_LLM_PROVIDER=openai with HINDSIGHT_API_LLM_BASE_URL pointing to Mistral's API (https://api.mistral.ai/v1), Hindsight crashes on startup during verify_connection because max_completion_tokens is sent but Mistral only accepts max_tokens, returning a 422:

openai.UnprocessableEntityError: Error code: 422 - {'message': {'detail': [{'type': 'extra_forbidden', 'loc': ['body', 'max_completion_tokens'], ...}]}}
RuntimeError: Connection verification failed for openai/mistral-small-latest

Solution

Add _max_tokens_param_name() method that selects the correct parameter name based on provider and base URL:

  • Native OpenAI (no custom base_url) → max_completion_tokens
  • Groqmax_completion_tokens (Groq supports the newer name)
  • openai with custom base_url, ollama, lmstudio, minimax, volcano → max_tokens

A custom base_url on the openai provider signals a third-party compatible API (Mistral, Together AI, etc.) that may not have adopted max_completion_tokens.

Testing

  • Existing tests pass unchanged (they test the call() argument name, not the API key)
  • The fix covers both call() and call_with_tools()

Mistral (and several other providers) reject 'max_completion_tokens' with a 422
because they haven't adopted the newer OpenAI parameter name. When the openai
provider is configured with a custom base_url (e.g. Mistral, Together AI),
fall back to the widely-supported 'max_tokens' parameter.

Native OpenAI (no custom base_url) and Groq still use 'max_completion_tokens'.

Fixes vectorize-io#852
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI provider sends max_completion_tokens — breaks Mistral (and likely other) OpenAI-compatible endpoints

1 participant