LLAMA Support?

[GINNOV]

this is fantastic to see happening. For experimental and cost management, it would be awesome to enable open source models (mlx friendly)

[AdonaiVera]

Hi @GINNOV 👋
I've been working on integrating multiple model options in my branch. The current integration supports Anthropic and GPT4All, with embeddings provided by OpenAI.

Anthropic works great, but they currently don't offer an embedding algorithm. As a workaround, I use embeddings from OpenAI. I’m also planning to integrate embeddings from Hugging Face soon to make the system more flexible and cost-efficient.
GPT4All runs well but is noticeably slow when running locally. However, it’s a great option for leveraging multiple models depending on the computational capacity available. You can use llama3.1 there.

I'd love to hear your thoughts or suggestions for further improvement! 🚀

[GINNOV]

thank you for the details I appreciate it. I will give it a go with gpt4all.
I value greatly the local use (no internet required) approach.

Happy to share my experience once I got it to working locally - keep cranking! Thanks again.