Leverage open-source, local LLMs via Ollama

How to leverage open-source, local LLMs via Ollama

This workflow shows how to leverage (i.e., authenticate, connect and prompt) an LLM (e.g., llama 3-instruct) available via Ollama in KNIME.

This approach is suitable for chat, instruct and code models.

____

Why do we use the OpenAI nodes to connect and prompt LLMs via Ollama?

Since Feb 2024, Ollama has built-in compatibility with the OpenAI Chat Completions API, making it possible to use more tooling and applications with Ollama locally. For this reason, we can conveniently use the OpenAI nodes of the KNIME AI Extension (and in particular the OpenAI Chat Model Connector node) to connect to the LLM of choice and send POST requests to the URL of a local host. In this way, we are freed from the burden of having to assemble and properly format the request body, and parse the resulting JSON output.

External resources

• OpenAI compatibility with Ollama
• Llama 3 model card on Ollama
• Download Ollama