First Contact with Genkit and Docker Model Runtime

It's no secret that I have multiple "technological" addictions. Particularly Generative AI (mainly with Docker Model Runner) and the Go language. Last night, I discovered Genkit Go which is a Go SDK from Google to facilitate the development of generative AI applications. I just had to test it! So, let's go!

Simple Completion

I started by initializing my program:

go mod init dmr-genkit-simple-completion
touch main.go
go get github.com/firebase/genkit/go
go mod tidy

And here's my first attempt, I ask the model to generate an NPC name for D&D:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/compat_oai/openai"
    "github.com/openai/openai-go/option"
)

func main() {
    ctx := context.Background()
    g := genkit.Init(ctx, genkit.WithPlugins(&openai.OpenAI{
        APIKey: "tada",
        Opts: []option.RequestOption{
            option.WithBaseURL("http://localhost:12434/engines/v1/"),
        },
    }))

    resp, err := genkit.Generate(ctx, g,
        ai.WithModelName("openai/ai/qwen2.5:0.5B-F16"),
        ai.WithMessages(
            ai.NewSystemTextMessage("You are the dungeon master of a D&D game."),
            ai.NewUserTextMessage("Generate a D&D NPC name."),
        ),
        ai.WithConfig(map[string]any{"temperature": 0.7}),
    )
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(resp.Text())
}

A few explanations:

• Docker Model Runner exposes an OpenAI-compatible API, so I use Genkit's openai.OpenAI plugin to connect to it.
• You need to provide an API key, but since Docker Model Runner doesn't require one, I put a dummy value.
• You also need to specify the API URL, here http://localhost:12434/engines/v1/ (default port).
• To define the model to use, you need to prefix its name with openai/.
• 🤚 don't forget to load the model with the command docker model pull ai/qwen2.5:0.5B-F16

Then, just run the program:

go run main.go

And here's the type of result you'll get:

"Shadow Weaver"

I'll let you adapt the code and start playing with it.

Now, let's move on to streaming completion!

Streaming Completion

The code is barely more complicated. This time, I'll ask the model to generate a complete NPC name with all its characteristics. Since the response might be long, I'll use streaming to display the response as it arrives.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/compat_oai/openai"
    "github.com/openai/openai-go/option"
)

func main() {
    ctx := context.Background()
    g := genkit.Init(ctx, genkit.WithPlugins(&openai.OpenAI{
        APIKey: "tada",
        Opts: []option.RequestOption{
            option.WithBaseURL("http://localhost:12434/engines/v1/"),
        },
    }))

    _, err := genkit.Generate(ctx, g,
        ai.WithModelName("openai/ai/qwen2.5:0.5B-F16"),

        ai.WithMessages(
            ai.NewSystemTextMessage("You are the dungeon master of a D&D game."),
            ai.NewUserTextMessage("Generate a D&D NPC Elf name and all its characteristics."),
        ),
        ai.WithConfig(map[string]any{"temperature": 0.7}),

        ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error {
            // Do something with the chunk...
            fmt.Print(chunk.Text())
            return nil
        }),
    )
    if err != nil {
        log.Fatal(err)
    }

}

And here's the type of result you'll get (displayed progressively):

**Elf NPC Name:** Elvish Guardian

### Characteristics:
- **Height:** 5'9" (approximately 175 cm)
- **Weight:** 200 lbs (approximately 90 kg)
- **Hair Color:** Light brown, with a silver sheen
- **Eyes:** Deep blue, with a glint
- **Hair Type:** Coarse, straight, and slightly wavy
- **Clothing:** Enthroned in a simple yet elegant robe, adorned with silver jewelry and a silver helmet and shield
- **Appearance:** Elvish, with a dignified and commanding demeanor
- **Combat Style:** Known for their martial prowess and ability to be both powerful and gentle
- **Abilities:** 10d6 Strength, 12d6 Dexterity, 12d6 Constitution, 12d6 Charisma, 12d6 Wisdom
- **Skills:** Combat, Athletics, Stealth, Perception, Investigation
- **Languages:** Elvish, Common
- **Background:** Elven Ranger, Elven Knight, Elven Mage

### Abilities:
- **Eldritch Shield:** Elvish magic that grants allies a +4 bonus to their next attack rolls against hostile creatures.
- **Vigilance:** Elvish magic that increases the chance of seeing enemies, allowing them to make Stealth checks.
- **Eldritch Bond:** Elvish magic that grants allies a +2 bonus to their next attack rolls against hostile creatures.
- **Thorn Touch:** Elvish magic that grants allies a +1 bonus to their next attack rolls against hostile creatures.
- **Mystic Shield:** Elvish magic that grants allies a +3 bonus to their next attack rolls against hostile creatures.
- **Frost Breath:** Elvish magic that grants allies a +2 bonus to their next attack rolls against hostile creatures.
- **Vortex Shield:** Elvish magic that grants allies a +4 bonus to their next attack rolls against hostile creatures.
- **Eldritch Wraith:** Elvish magic that grants allies a +5 bonus to their next attack rolls against hostile creatures.
- **Arcane Shield:** Elvish magic that grants allies a +3 bonus to their next attack rolls against hostile creatures.

### Background:
Elvish Guardians are legendary warriors and protectors of the land of Eldoria. They are known for their strength, wisdom, and their ability to communicate through the senses and use the elements of their natural form to their advantage. Elvish Guardians are respected for their loyalty, courage, and their ability to bring peace and harmony to the land

I kept it simple in my prompts and I'm using a relatively small model (0.5B). Feel free to make the prompts more complex and test with larger models.

Come on, one last example to close this introduction: function calling (which nowadays is essential when you want to develop AI agents).

Function Calling

For this last example I'm going to change models and use hf.co/menlo/jan-nano-gguf:q4_k_m which handles function calling quite well. I'm going to create two "tools": one to roll dice and another to generate non-player character names in the Dungeons and Dragons universe. The model will be able to call these functions based on user requests.

package main

import (
    "context"
    "fmt"
    "math/rand"
    "strings"
    "time"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/compat_oai/openai"
    "github.com/openai/openai-go/option"
)

type DiceRollInput struct {
    NumDice int `json:"num_dice"`
    NumFaces int `json:"num_faces"`
}

type DiceRollResult struct {
    Rolls []int `json:"rolls"`
    Total int   `json:"total"`
}

type CharacterNameInput struct {
    Race string `json:"race"`
}

type CharacterNameResult struct {
    Name string `json:"name"`
    Race string `json:"race"`
}

func main() {
    ctx := context.Background()
    g := genkit.Init(ctx, genkit.WithPlugins(&openai.OpenAI{
        APIKey: "tada",
        Opts: []option.RequestOption{
            option.WithBaseURL("http://localhost:12434/engines/v1/"),
        },
    }))

    // Define tools
    diceRollTool := genkit.DefineTool(g, "roll_dice", "Roll n dice with n faces each",
        func(ctx *ai.ToolContext, input DiceRollInput) (DiceRollResult, error) {
            return rollDice(input.NumDice, input.NumFaces), nil
        },
    )

    characterNameTool := genkit.DefineTool(g, "generate_character_name", "Generate a D&D character name for a specific race",
        func(ctx *ai.ToolContext, input CharacterNameInput) (CharacterNameResult, error) {
            return generateCharacterName(input.Race), nil
        },
    )

    // Create system message
    systemMsg := ai.NewSystemTextMessage(`
        You are a helpful D&D assistant that can roll dice and generate character names.
        Use the appropriate tools when asked to roll dice or generate character names.
        `,
    )

    // Create user message
    userMsg := ai.NewUserTextMessage(`
        Roll 3 dice with 6 faces each. 
        Then generate a character name for an elf.
        Finally, roll 2 dice with 8 faces each.
        After that, generate a character name for a dwarf.
        `,
    )

    resp, err := genkit.Generate(ctx, g,
        ai.WithModelName("openai/hf.co/menlo/jan-nano-gguf:q4_k_m"),

        ai.WithMessages(systemMsg, userMsg),
        ai.WithTools(diceRollTool, characterNameTool),
        ai.WithToolChoice(ai.ToolChoiceAuto),
    )
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return
    }

    fmt.Println(resp.Text())
}

func rollDice(numDice, numFaces int) DiceRollResult {
    r := rand.New(rand.NewSource(time.Now().UnixNano()))
    rolls := make([]int, numDice)
    total := 0

    for i := 0; i < numDice; i++ {
        roll := r.Intn(numFaces) + 1
        rolls[i] = roll
        total += roll
    }

    return DiceRollResult{
        Rolls: rolls,
        Total: total,
    }
}

func generateCharacterName(race string) CharacterNameResult {
    r := rand.New(rand.NewSource(time.Now().UnixNano()))

    namesByRace := map[string][]string{
        "elf": {"Aerdrie", "Ahvonna", "Aramil", "Aranea", "Berrian", "Caelynn", "Carric", "Dayereth", "Enna", "Galinndan"},
        "dwarf": {"Adrik", "Baern", "Darrak", "Eberk", "Fargrim", "Gardain", "Harbek", "Kildrak", "Morgran", "Thorek"},
        "human": {"Aerdrie", "Aramil", "Berris", "Cithreth", "Dayereth", "Enna", "Galinndan", "Hadarai", "Immeral", "Lamlis"},
        "halfling": {"Alton", "Ander", "Bernie", "Bobbin", "Cade", "Callus", "Corrin", "Dannad", "Garret", "Lindal"},
        "orc": {"Gash", "Gell", "Henk", "Holg", "Imsh", "Keth", "Krusk", "Mhurren", "Ront", "Shump"},
        "tiefling": {"Akmenos", "Amnon", "Barakas", "Damakos", "Ekemon", "Iados", "Kairon", "Leucis", "Melech", "Mordai"},
    }

    raceLower := strings.ToLower(race)
    names, exists := namesByRace[raceLower]
    if !exists {
        names = namesByRace["human"] // Default to human names
    }

    selectedName := names[r.Intn(len(names))]

    return CharacterNameResult{
        Name: selectedName,
        Race: race,
    }
}

Defining a "tool" is pretty simple with Genkit. You just need to create a structure for the inputs and another for the outputs:

type DiceRollInput struct {
    NumDice int `json:"num_dice"`
    NumFaces int `json:"num_faces"`
}

type DiceRollResult struct {
    Rolls []int `json:"rolls"`
    Total int   `json:"total"`
}

Then provide a function that takes the input structure as input and returns the output structure:

diceRollTool := genkit.DefineTool(g, "roll_dice", "Roll n dice with n faces each",
    func(ctx *ai.ToolContext, input DiceRollInput) (DiceRollResult, error) {
        return rollDice(input.NumDice, input.NumFaces), nil
    },
)

Then I created a user message that asks the model to perform several actions (roll dice and generate character names) to test the model's ability to call multiple functions from a single prompt:

// Create user message
userMsg := ai.NewUserTextMessage(`
    Roll 3 dice with 6 faces each. 
    Then generate a character name for an elf.
    Finally, roll 2 dice with 8 faces each.
    After that, generate a character name for a dwarf.
    `,
)

When you run the program, you should get a result like:

The results of your actions are as follows:

1. You rolled 3 dice with 6 faces each, resulting in [1, 2, 6] (total = 9).
2. The generated character name for an elf is **Aranea**.
3. You rolled 2 dice with 8 faces each, resulting in [8, 4] (total = 12).
4. The generated character name for a dwarf is **Fargrim**.

And that's it for this quick introduction to Genkit with Docker Model Runner. I hope this will make you want to test these tools yourself. Have fun! And see you soon for another Genkit discovery article.