issues with WorkFlows and local llms

[ehsankf]

Hi,

When running the workflow in this tutorial:
https://docs.llamaindex.ai/en/stable/getting_started/starter_example_local/

it runs to the following error:
llama_index.core.workflow.errors.WorkflowRuntimeError: Error in step 'run_agent_step': LLM must be a FunctionCallingLLM

[marcusschiesser]

Which Ollama model are you using? They are not all supporting function calls.

[ehsankf]

I used the same model as in the tutorial: "llama3.1". I got the response below from the agent workflow for the query "What is 1234 * 4567?".

{"name": "multiply", "parameters": {"a": 1234, "b": 4567}}

[dosubot[bot]]

The error WorkflowRuntimeError: Error in step 'run_agent_step': LLM must be a FunctionCallingLLM occurs because the workflow expects the LLM you provide to support OpenAI-style function calling. In LlamaIndex, this means your LLM instance must implement the FunctionCallingLLM interface, which requires specific methods for handling tool/function calls in agent workflows. This is especially relevant when using the FunctionAgent class, as shown in the tutorial.

For local models like Ollama, make sure you are using the latest version of the llama-index-llms-ollama package and that your Ollama model is initialized with is_function_calling_model=True to indicate it supports function calling. There have been recent updates and bug fixes in this area, so updating your packages can resolve many issues. You can do this with:

pip install -U llama-index llama-index-core llama-index-llms-ollama

If you are using a different local LLM, ensure it implements the required methods (get_tool_calls_from_response and _prepare_chat_with_tools). You can find working examples in the Ollama and Anthropic integrations for reference on how to implement these methods if you are building a custom LLM class source.

If you are still seeing the error after updating, double-check that your LLM initialization in the tutorial script matches the expected pattern for function calling. For Ollama, it should look something like:

from llama_index.llms.ollama import Ollama

llm = Ollama(
    model="your-model-name",
    is_function_calling_model=True,
    # ...other params
)

source

If you are using a different agent or workflow, or a non-OpenAI LLM, you may need to use a different agent class (such as ReActAgent) or update your workflow to avoid features that require function calling support source.

Let me know if you need help with a specific model or code snippet!

To reply, just mention my name @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}