Add Support for Local LLM Models

[galasissy]

Description

I'm currently using LightAgent and appreciate its lightweight design and flexibility. However, I would find it extremely valuable if there was support for local LLM models (like Llama, Mistral, etc.) that could be run without needing external API calls.

Use Case

Many users, including myself, want to:

• Run agents completely offline for privacy concerns
• Reduce API costs associated with commercial models
• Experiment with open-source models through local inference

Proposed Implementation

Perhaps this could be implemented by:

1. Adding support for local inference libraries like llama.cpp or vLLM
2. Creating an adapter interface that allows interchangeability between cloud and local models
3. Including documentation on hardware requirements and optimization tips

Additional Context

I've seen similar functionality in other frameworks, and this would make LightAgent even more versatile while maintaining its core lightweight philosophy.

Would this be something you'd consider adding to the roadmap?

[wxai-space]

The local inference addresses generated after deploying llama.cpp and vLLM depend on the specific deployment method and configuration. For example, after running llama.cpp, the following inference address is generated:
Chat completion endpoint: http://localhost:8080/v1/chat/completions

• llama.cpp: The default address is http://localhost:8080/v1 (Web service mode).
• vLLM: The default address is http://localhost:8000/v1 (API service).

LightAgent supports filling in the above addresses as the base_url, and the api_key can be any value, allowing the agent to run normally.

---

Thank you for your question! If you have any further inquiries, feel free to ask. 😊