Function Calling with Ollama and LLMs that do not support function calling

In a previous blog post, I explained "Function Calling" and how to use it with Mistral 7B, which implements this feature. You can read it here (it's a prerequisite for reading the present blog post): Function Calling with Ollama, Mistral 7B, Bash and Jq.

Nevertheless, it is possible to reproduce this feature with some LLMs that do not implement the "Function Calling" feature natively, but we need to supervise them and explain precisely what we need. The result (the output) will be less predictable, so you will need to add some tests before using the output, but with some "clever" LLMs, you will obtain correct results.

Let's see how to implement this. I will use the Phi3:mini LLM, which seems very promising for this experiment.

A "Function Calling" prompt

I did my experiments with Bash, Jq and Curl. Usually, a function calling prompt looks like this:

read -r -d '' TOOLS_CONTENT <<- EOM
[AVAILABLE_TOOLS]
[
    {
        "type": "function", 
        "function": {
            "name": "hello",
            "description": "Say hello to a given person with his name",
            "parameters": {
                "type": "object", 
                "properties": {
                    "name": {
                        "type": "string", 
                        "description": "The name of the person"
                    }
                }, 
                "required": ["name"]
            }
        }
    },
    {
        "type": "function", 
        "function": {
            "name": "addNumbers",
            "description": "Make an addition of the two given numbers",
            "parameters": {
                "type": "object", 
                "properties": {
                    "a": {
                        "type": "number", 
                        "description": "first operand"
                    },
                    "b": {
                        "type": "number",
                        "description": "second operand"
                    }
                }, 
                "required": ["a", "b"]
            }
        }
    }
]
[/AVAILABLE_TOOLS]
EOM

USER_CONTENT='[INST] say "hello" to Bob [/INST]'

And I can send it to Ollama thanks to a JSON payload like this one:

read -r -d '' DATA <<- EOM
{
  "model":"${MODEL}",
  "options": {
    "temperature": 0.0,
    "repeat_last_n": 2
  },
  "messages": [
    {"role":"system", "content": "${TOOLS_CONTENT}"},
    {"role":"user", "content": "${USER_CONTENT}"}
  ],
  "stream": false,
  "format": "json",
  "raw": true
}
EOM

If I try the script I did in the previous blog post with Phi3:mini LLM, it won't work because Phi3 does not implement the function calling feature and doesn't understand what I want. So, I need to give it more instructions.

A supervised "Function Calling" prompt

For that, I will start with an introductory message:

read -r -d '' SYSTEM_INTRODUCTION <<- EOM
You have access to the following tools:
EOM

And I will finish with a set of detailed instructions to explain what to do, where to find the information, how to format the result, ...

read -r -d '' SYSTEM_INSTRUCTIONS <<- EOM
If the question of the user matched the description of a tool, 
the tool will be called.

To call a tool, respond with a JSON object with the following structure: 
{
  "name": <name of the called tool>,
  "arguments": {
    <name of the argument>: <value of the argument>
  }
}

search the name of the tool in the list of tools with the Name field
EOM

At the end, the JSON payload to be sent to Ollama will look like this:

read -r -d '' DATA <<- EOM
{
  "model":"${MODEL}",
  "options": {
    "temperature": 0.0,
    "repeat_last_n": 2
  },
  "messages": [
    {"role":"system", "content": "${SYSTEM_INTRODUCTION}"},
    {"role":"system", "content": "${TOOLS_CONTENT}"},
    {"role":"system", "content": "${SYSTEM_INSTRUCTIONS}"},
    {"role":"user", "content": "${USER_CONTENT}"}
  ],
  "stream": false,
  "format": "json",
  "raw": true
}
EOM

You can see that, I had surrounded the TOOLS_CONTENT with SYSTEM_INTRODUCTION and SYSTEM_INSTRUCTIONS

The final script

The final script is almost the same as the previous one (see the previous blog post: Function Calling with Ollama, Mistral 7B, Bash and Jq)

#!/bin/bash

# Generate a completion
function Chat() {
    OLLAMA_URL="${1}"
    DATA="${2}"

    JSON_RESULT=$(curl --silent ${OLLAMA_URL}/api/chat \
        -H "Content-Type: application/json" \
        -d "${DATA}"
    )
    echo "${JSON_RESULT}"
}

# Remove any newline characters (\n)
function Sanitize() {
    CONTENT="${1}"
    CONTENT=$(echo ${CONTENT} | tr -d '\n')
    echo "${CONTENT}"
}

# Escapes double quotes in the given content
function EscapeDoubleQuotes() {
    CONTENT="${1}"
    CONTENT=$(echo ${CONTENT} | sed 's/"/\\"/g')
    echo "${CONTENT}"
}

OLLAMA_URL=${OLLAMA_URL:-http://localhost:11434}
MODEL="phi3:mini"

# The list of tools
read -r -d '' TOOLS_CONTENT <<- EOM
[AVAILABLE_TOOLS]
[
    {
        "type": "function", 
        "function": {
            "name": "hello",
            "description": "Say hello to a given person with his name",
            "parameters": {
                "type": "object", 
                "properties": {
                    "name": {
                        "type": "string", 
                        "description": "The name of the person"
                    }
                }, 
                "required": ["name"]
            }
        }
    },
    {
        "type": "function", 
        "function": {
            "name": "addNumbers",
            "description": "Make an addition of the two given numbers",
            "parameters": {
                "type": "object", 
                "properties": {
                    "a": {
                        "type": "number", 
                        "description": "first operand"
                    },
                    "b": {
                        "type": "number",
                        "description": "second operand"
                    }
                }, 
                "required": ["a", "b"]
            }
        }
    }
]
[/AVAILABLE_TOOLS]
EOM
# Format the string
TOOLS_CONTENT=$(EscapeDoubleQuotes "${TOOLS_CONTENT}")
TOOLS_CONTENT=$(Sanitize "${TOOLS_CONTENT}")

# The introduction message for the AI agent
read -r -d '' SYSTEM_INTRODUCTION <<- EOM
You have access to the following tools:
EOM
SYSTEM_INTRODUCTION=$(Sanitize "${SYSTEM_INTRODUCTION}")

# The detailed instructions for the AI agent
read -r -d '' SYSTEM_INSTRUCTIONS <<- EOM
If the question of the user matched the description of a tool, 
the tool will be called.

To call a tool, respond with a JSON object with the following structure: 
{
  "name": <name of the called tool>,
  "arguments": {
    <name of the argument>: <value of the argument>
  }
}

search the name of the tool in the list of tools with the Name field
EOM
SYSTEM_INSTRUCTIONS=$(EscapeDoubleQuotes "${SYSTEM_INSTRUCTIONS}")
SYSTEM_INSTRUCTIONS=$(Sanitize "${SYSTEM_INSTRUCTIONS}")

# The user question
USER_CONTENT='[INST] say "hello" to Bob [/INST]'
USER_CONTENT=$(EscapeDoubleQuotes "${USER_CONTENT}")

# The JSON payload
read -r -d '' DATA <<- EOM
{
  "model":"${MODEL}",
  "options": {
    "temperature": 0.0,
    "repeat_last_n": 2
  },
  "messages": [
    {"role":"system", "content": "${SYSTEM_INTRODUCTION}"},
    {"role":"system", "content": "${TOOLS_CONTENT}"},
    {"role":"system", "content": "${SYSTEM_INSTRUCTIONS}"},
    {"role":"user", "content": "${USER_CONTENT}"}
  ],
  "stream": false,
  "format": "json",
  "raw": true
}
EOM

# Speak to Ollama and Phi3:mini
jsonResult=$(Chat "${OLLAMA_URL}" "${DATA}")
messageContent=$(echo "${jsonResult}" | jq -r '.message.content')
messageContent=$(Sanitize "${messageContent}")
echo "${messageContent}"

When you run the script, you will get this:

{"name":"hello","arguments":{"name":"Bob"}}

And you can try with this user question:

USER_CONTENT='[INST] add 2 and 40 [/INST]'

And you will get this:

{"name":"addNumbers","arguments":{"a":2,"b":40}}

Be careful

Doing this with LLMs that do not implement the function calling feature natively can't guarantee the predictability of the results, so before using the output, check the format of the result and retry if the format is not valid.

This "trick" won't work with all the LLMs, and you will need to adapt the instructions.

Enjoy hacking Ollama and LLMs!