Skip to content

Implement use of custom OpenAI API endpoints #19

@hillct

Description

@hillct

By adding an OpenAI API endpoint URL field, in addition to the existing API key field in settings, support for LMStudio, Ollama and vLLM the three most common local model runners. This approach would aupercede Ticket #15 which would suggests implementing the custom Ollama API, while Ollama as well as the other two mentioned model runners all support the OpenAI API as well. Suxh an implementation should allow for the full base endpoint to be specified, because inevitably sime deployments will use https, and others http only, some will be on nonstandard ports or be terminated as a nonstandard path. Do not assume when we speak of local LLMs, that we mean only localhost, as many users will have separate or even multiple separate servers where model runners are deployed (so supporting multiple instances of openI API servers will be helpful as well, though supporting one with custom bnase URL is a great starting point. In such an implementation the API key should be considered optional, used if provided, but if a custom bae URL is provided, it should be optional. Of course, if no custom base URL is provided, then the OpenAI standard baqse URL should be used and the API key would be required (if Openai API usage is enabled). Also, it would be a significant enhancement to cause the model list endpoint to be called, in the case of each API base URL, to generate a complete and dynamic list of models accessible using that API base instance, rather than providing a fixed list in the setting pane. This woul be a bit more work, but again a significant improvement to both UI and application versitility. When refreshed, the listy would be rendered, and the user would have the option to enable or disable any set of models they preferred. This woul also reduce the requirement to spin new releases each time new models are added by any provider.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions