Ollama Support?

[rhyswynn]

I created an ollama.py script that handles an Ollama client and it works fine as an integration. It does require a few other file changes. Any thoughts about adding support for Ollama? I thought Ollama was better than individual local LLM support because many different versions can all be supported with one class. Thank you!

[Holmeswww]

Thank you for your active support!

ollama does seem like a good one. Depending on the changes, I'm considering adding this as an optional pip flag pip install agentkit-llm[ollama]

Would you offer more info on the changes involved?

[rhyswynn]

You can see the changes in my fork. The changes in GPT.py were already updated in your repo, the base_node change is just a personal preference. The Ollama changes are all in the llm_api folder, the init and utils get updated, and the new ollama.py. I added 2 new optional parameters to the get_query function, but it could work fine with environment variables, following the pattern of the Azure and OpenAI details. Thank you!
main...rhyswynn:AgentKit:main

[rhyswynn]

When OIlama downloads a model, it downloads a tokenizer json and config, but I could not find a way to instantiate a tokenizer using those files. That would be a preferred approach. I manually downloaded the llama3 model and that comes with a tokenizer.model file that works with the way I implemented it. I added some documentation to the readme in a version of this module when I was considering maintaining and developing another version, before I understood how it works fully. https://github.com/rhyswynn/AgentKit/blob/dev/README.md#built-in-ollama-api

[Holmeswww]

Thanks for the clarifications.
This seems like an exciting addition to the framework. Let me try to add it.

[Holmeswww]

https://github.com/Holmeswww/AgentKit/releases/tag/v0.1.7

[rhyswynn]

It seems you carried over some of the function parameters from my example, they need to be removed. There are some other minor changes that need to be fixed so that it works. I synced my fork with your latest version and made only those changes needed to make it work. main...rhyswynn:AgentKit:main

[rhyswynn]

Also, there are some much larger token models available, and an additional parameter that is needed in some cases. llama3-gradient for example is a 1M token model. I asked a very complex question and it never gave me an answer, I think because it was still generating tokens and I ran out of memory maybe as it needs 64GB of vram for 250k tokens or something like that, I forget the numbers. Anyway, the additional option is num_ctx and it sets the context window in number of tokens. I set this to 32k to allow the model to work on my system. Maybe another 2 environment properties that can override the default 8192 and no specified context window. Thank you for keeping this package updated!
{"messages": messages, "model": "llama3-gradient", "options":{"num_ctx": 32000}, "stream": False}