llm

package module
v0.0.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 11, 2025 License: Apache-2.0 Imports: 12 Imported by: 0

README

go-llm

The module implements a simple API interface for large language models which run on Ollama, Anthopic, Mistral and OpenAI. The module implements the ability to:

  • Maintain a session of messages;
  • Tool calling support, including using your own tools (aka Tool plugins);
  • Create embedding vectors from text;
  • Stream responses;
  • Multi-modal support (aka, Images, Audio and Attachments);
  • Text-to-speech (OpenAI only) for completions

There is a command-line tool included in the module which can be used to interact with the API. If you have docker installed, you can use the following command to run the tool, without installation:

# Display version, help
docker run ghcr.io/mutablelogic/go-llm version
docker run ghcr.io/mutablelogic/go-llm --help

# Interact with Claude to retrieve news headlines, assuming
# you have an API key for both Anthropic and NewsAPI
docker run -e ANTHROPIC_API_KEY -e NEWSAPI_KEY \
  ghcr.io/mutablelogic/go-llm \
  chat mistral-small-latest --prompt "What is the latest news?"

See below for more information on how to use the command-line tool (or how to install it if you have a go compiler).

Programmatic Usage

See the documentation here for integration into your own code.

Agent Instantiation

For each LLM provider, you create an agent which can be used to interact with the API. To create an Ollama agent,

import (
  "github.com/mutablelogic/go-llm/pkg/ollama"
)

func main() {
  // Create a new agent - replace the URL with the one to your Ollama instance
  agent, err := ollama.New("https://ollama.com/api/v1/")
  if err != nil {
    panic(err)
  }
  // ...
}

To create an Anthropic agent with an API key stored as an environment variable,

import (
  "github.com/mutablelogic/go-llm/pkg/anthropic"
)

func main() {
  // Create a new agent
  agent, err := anthropic.New(os.Getenv("ANTHROPIC_API_KEY"))
  if err != nil {
    panic(err)
  }
  // ...
}

For Mistral models, you can use:

import (
  "github.com/mutablelogic/go-llm/pkg/mistral"
)

func main() {
  // Create a new agent
  agent, err := mistral.New(os.Getenv("MISTRAL_API_KEY"))
  if err != nil {
    panic(err)
  }
  // ...
}

Similarly for OpenAI models, you can use:

import (
  "github.com/mutablelogic/go-llm/pkg/openai"
)

func main() {
  // Create a new agent
  agent, err := openai.New(os.Getenv("OPENAI_API_KEY"))
  if err != nil {
    panic(err)
  }
  // ...
}

You can append options to the agent creation to set the client/server communication options, such as user agent strings, timeouts, debugging, rate limiting, adding custom headers, etc. See here for more information.

There is also an aggregated agent which can be used to interact with multiple providers at once. This is useful if you want to use models from different providers simultaneously.

import (
  "github.com/mutablelogic/go-llm/pkg/agent"
)

func main() {
  // Create a new agent which aggregates multiple providers
  agent, err := agent.New(
    agent.WithAnthropic(os.Getenv("ANTHROPIC_API_KEY")), 
    agent.WithMistral(os.Getenv("MISTRAL_API_KEY")),
    agent.WithOpenAI(os.Getenv("OPENAI_API_KEY")),
    agent.WithOllama(os.Getenv("OLLAMA_URL")),
  )
  if err != nil {
    panic(err)
  }
  // ...
}
Completion

You can generate a completion as follows,

import (
  "github.com/mutablelogic/go-llm"
)

func completion(ctx context.Context, agent llm.Agent) (string, error) {
  completion, err := agent.
    Model(ctx, "claude-3-5-haiku-20241022").
    Completion((ctx, "Why is the sky blue?")
  if err != nil {
    return "", err
  } else {
    return completion.Text(0), nil
  }
}

The zero index argument on completion.Text(int) indicates you want the text from the zero'th completion choice, for providers who can generate serveral different choices simultaneously.

Use one of the following options as an argument to the Completion method to customize the output format of the completion, which needs to be paired with the right model:

  • llm.WithFormat("text") - Generate text output (default).
  • llm.WithFormat("json") - Generate JSON output.
  • llm.WithFormat("image", "jpeg") - Generate JPEG image output (for models which support it).
  • llm.WithFormat("audio", "mp3") - Generate audio output (for models which support it). Possible values are mp3, opus, aac, flac, wav, and pcm.
Chat Sessions

You create a chat session with a model as follows,

import (
  "github.com/mutablelogic/go-llm"
)

func session(ctx context.Context, agent llm.Agent) error {
  // Create a new chat session
  session := agent.
    Model(ctx, "claude-3-5-haiku-20241022").
    Context()

  // Repeat forever
  for {
    err := session.FromUser(ctx, "hello")
    if err != nil {
      return err
    }

    // Print the response for the zero'th completion
    fmt.Println(session.Text(0))
  }
}

The Context object will continue to store the current session and options, and will ensure the session is maintained across multiple completion calls.

Embedding Generation

You can generate embedding vectors using an appropriate model with Ollama, OpenAI and Mistral models:

import (
  "github.com/mutablelogic/go-llm"
)

func embedding(ctx context.Context, agent llm.Agent) error {
  vector, err := agent.
    Model(ctx, "mistral-embed").
    Embedding(ctx, "hello")
  // ...
}
Attachments & Image Caption Generation

Some models have vision capability and others can also summarize text. For example, to generate captions for an image,

import (
  "github.com/mutablelogic/go-llm"
)

func generate_image_caption(ctx context.Context, agent llm.Agent, path string) (string, error) {
  f, err := os.Open(path)
  if err != nil {
    return "", err
  }
  defer f.Close()

  completion, err := agent.
    Model(ctx, "claude-3-5-sonnet-20241022").
    Completion((ctx, "Provide a short caption for this image", llm.WithAttachment(f))
  if err != nil {
    return "", err  
  }    

  return completion.Text(0), nil
}

To summarize a text or PDF document is exactly the same using an Anthropic model, but maybe with a different prompt.

Streaming

Streaming is supported with all providers, but Ollama cannot be used with streaming and tools simultaneously. You provide a callback function of signature func(llm.Completion) which will be called as a completion is received.

import (
  "github.com/mutablelogic/go-llm"
)

func generate_completion(ctx context.Context, agent llm.Agent, prompt string) (string, error) {
   completion, err := agent.
    Model(ctx, "claude-3-5-haiku-20241022").
    Completion((ctx, "Why is the sky blue?", llm.WithStream(stream_callback))
  if err != nil {
    return "", err
  } else {
    return completion.Text(0), nil
  }
}

func stream_callback(completion llm.Completion) {
  // Print out the completion text on each call
  fmt.Println(completion.Text(0))
}
Tool Support

All providers support tools, but not all models. Your own tools should implement the following interface:

package llm

// Definition of a tool
type Tool interface {
  Name() string                     // The name of the tool
  Description() string              // The description of the tool
  Run(context.Context) (any, error) // Run the tool with a deadline and 
                                    // return the result
}

For example, if you want to implement a tool which adds two numbers,

package addition

type Adder struct {
  A float64 `name:"a" help:"The first number" required:"true"`
  B float64 `name:"b" help:"The second number" required:"true"`
}

func (Adder) Name() string {
  return "add_two_numbers"
}

func (Adder) Description() string {
  return "Add two numbers together and return the result"
}

func (a Adder) Run(context.Context) (any, error) {
  return a.A + a.B, nil
}

Then you can include your tool as part of the completion. It's possible that a completion will continue to call additional tools, in which case you should actually loop through completions until no tool calls are made.

import (
  "github.com/mutablelogic/go-llm"
  "github.com/mutablelogic/go-llm/pkg/tool"
)

func add_two_numbers(ctx context.Context, agent llm.Agent) (string, error) {
  context := agent.Model(ctx, "claude-3-5-haiku-20241022").Context()
  toolkit := tool.NewToolKit()
  toolkit.Register(&Adder{})

  // Get the tool call
  if err := context.FromUser(ctx, "What is five plus seven?", llm.WithToolKit(toolkit)); err != nil {
    return "", err
  }

  // Call tools
  for {
    calls := context.ToolCalls(0)
    if len(calls) == 0 {
      break
    }

    // Print out any intermediate messages
    if context.Text(0) != "" {
      fmt.Println(context.Text(0))      
    }

    // Get the results from the toolkit
    results, err := toolkit.Run(ctx, calls...)
    if err != nil {
      return "", err
    }

    // Get another tool call or a user response
    if err := context.FromTool(ctx, results...); err != nil {
      return "", err
    }
  }

  // Return the result
  return context.Text(0)
}

Parameters are implemented as struct fields, with tags. The tags you can include are:

  • name:"" - Set the name for the parameter
  • json:"" - If name is not used, then the name is set from the json tag
  • help:": - Set the description for the parameter
  • required:"" - The parameter is required as part of the tool call
  • enum:"a,b,c" - The parameter value should be one of these comma-separated options

The transation of field types is as follows:

  • string - Translates as JSON string
  • uint, int - Translates to JSON integer
  • float32, float64 - Translates to JSON number

The Command Line Tool

You can use the command-line tool to interact with the API. To build the tool, you can use the following command:

go install github.com/mutablelogic/go-llm/cmd/llm@latest
llm --help

The output is something like:

Usage: llm <command> [flags]

LLM agent command line interface

Flags:
  -h, --help                      Show context-sensitive help.
      --debug                     Enable debug output
  -v, --verbose                   Enable verbose output
      --timeout=DURATION          Agent connection timeout
      --ollama-endpoint=STRING    Ollama endpoint ($OLLAMA_URL)
      --anthropic-key=STRING      Anthropic API Key ($ANTHROPIC_API_KEY)
      --mistral-key=STRING        Mistral API Key ($MISTRAL_API_KEY)
      --open-ai-key=STRING        OpenAI API Key ($OPENAI_API_KEY)
      --gemini-key=STRING         Gemini API Key ($GEMINI_API_KEY)
      --news-key=STRING           News API Key ($NEWSAPI_KEY)

Commands:
  agents       Return a list of agents
  models       Return a list of models
  tools        Return a list of tools
  download     Download a model (for Ollama)
  chat         Start a chat session
  complete     Complete a prompt, generate image or speech from text
  embedding    Generate an embedding
  version      Print the version of this tool

Run "llm <command> --help" for more information on a command.
Prompt Completion

To have the model respond to a prompt, you can use the complete command. For example, to have the model respond to the prompt "What is the capital of France?" using the claude-3-5-haiku-20241022 model, you can use the following command:

llm complete "What is the capital of France?"

The first time you use the command use the --model flag to specify the model you want to use. Your choice of model will be remembered for subsequent completions.

Explain computer code

To have the model explain a piece of computer code, you can pipe the code into the complete command. For example, to have the model explain the code in the file example.go, you can use the following command:

cat example.go | llm complete
Caption an image

To have the model generate a caption for an image, you can use the complete command with the --file flag. For example, to have the model generate a caption for the image in the file example.jpg, you can use the following command:

llm complete --file picture.png "Explain this image"
Generate an image

To have the model generate an image from a prompt, you can use the complete command with the --format image option. For example, to have the model generate an image from the prompt "A picture of a cat", you can use the following command:

llm complete --model dall-e-3 --format image "A picture of a cat"

Flags --size, --quality and --style can be used to specify the image parameters. It will write the image file in the current working directory.

Convert text to speech

To have a model generate text from speech:

echo book.txt | llm complete --model tts-1 --format mp3 --voice coral

It will write the audio file in the current working directory. You can currently write the following audio formats and voices:

  • Formats: --format mp3, --format opus, --format aac, --format flac, --format wav, --format pcm
  • Voices: --voice alloy, --voice ash, --voice coral, --voice echo, --voice fable, --voice onyx, --voice nova, --voice sage, --voice shimmer

Contributing & Distribution

This module is currently in development and subject to change. Please do file feature requests and bugs here. The license is Apache 2 so feel free to redistribute. Redistributions in either source code or binary form must reproduce the copyright notice, and please link back to this repository for more information:

go-llm
https://github.com/mutablelogic/go-llm/
Copyright (c) 2025 David Thorpe, All rights reserved.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Agent 

type Agent interface {
	// Return the name of the agent
	Name() string

	// Return the models
	Models(context.Context) ([]Model, error)

	// Return a model by name, or nil if not found.
	// Panics on error.
	Model(context.Context, string) Model
}

An LLM Agent is a client for the LLM service

type Attachment 

type Attachment struct {
	// contains filtered or unexported fields
}

Attachment for messages

func NewAttachment added in v0.0.6 

func NewAttachment() *Attachment

NewAttachment creates a new, empty attachment

func NewAttachmentWithImage added in v0.0.10 

func NewAttachmentWithImage(image *ImageMeta) *Attachment

NewAttachment with OpenAI image

func ReadAttachment 

func ReadAttachment(r io.Reader, mimetype ...string) (*Attachment, error)

ReadAttachment returns an attachment from a reader object. It is the responsibility of the caller to close the reader.

func (*Attachment) Append added in v0.0.6 

func (a *Attachment) Append(other *Attachment)

Streaming includes the ability to append data

func (*Attachment) Caption added in v0.0.6 

func (a *Attachment) Caption() string

Return the caption for the attachment

func (*Attachment) Data 

func (a *Attachment) Data() []byte

Return the raw attachment data

func (*Attachment) Filename 

func (a *Attachment) Filename() string

Return the filename of an attachment

func (*Attachment) Hash added in v0.0.10 

func (a *Attachment) Hash() string

Compute and print the MD5 hash

func (*Attachment) MarshalJSON added in v0.0.6 

func (a *Attachment) MarshalJSON() ([]byte, error)

Convert an attachment into JSON

func (*Attachment) String added in v0.0.3 

func (a *Attachment) String() string

Stringify an attachment

func (*Attachment) Type added in v0.0.3 

func (a *Attachment) Type() string

Return the mime media type for the attachment, based on the data and/or filename extension. Returns an empty string if there is no data or filename

func (*Attachment) UnmarshalJSON added in v0.0.6 

func (a *Attachment) UnmarshalJSON(data []byte) error

Convert JSON into an attachment

func (*Attachment) Url added in v0.0.3 

func (a *Attachment) Url() string

func (*Attachment) Write added in v0.0.10 

func (a *Attachment) Write(w io.Writer) (int, error)

Write out attachment

type AttachmentMeta added in v0.0.6 

type AttachmentMeta struct {
	Id        string `json:"id,omitempty"`
	Filename  string `json:"filename,omitempty"`
	ExpiresAt uint64 `json:"expires_at,omitempty"`
	Caption   string `json:"transcript,omitempty"`
	Data      []byte `json:"data"`
	Type      string `json:"type"`
}

General attachment metadata

type Completion added in v0.0.3 

type Completion interface {
	// Return the number of completions, which is ususally 1 unless
	// WithNumCompletions was used
	Num() int

	// Return a specific completion
	Choice(int) Completion

	// Return the current session role, which can be system, assistant,
	// user, tool, tool_result, ...
	// If this is a completion, the role is usually 'assistant'
	Role() string

	// Return the text for the last completion, with the argument as the
	// completion index (usually 0).
	Text(int) string

	// Return attachment (audio/image) for the last completion, with the
	// argument as the completion index (usually 0).
	Attachment(int) *Attachment

	// Return the current session tool calls given the completion index.
	// Will return nil if no tool calls were returned.
	ToolCalls(int) []ToolCall
}

Completion is the content of the last message

type Context 

type Context interface {
	Completion

	// Generate a response from a user prompt (with attachments and
	// other options)
	FromUser(context.Context, string, ...Opt) error

	// Generate a response from a tool, passing the results
	// from the tool call
	FromTool(context.Context, ...ToolResult) error

	// Return the duration since the last completion was made
	// or zero
	SinceLast() time.Duration
}

Context is a context window fed to the agent to generate a response, with the ability to create the next completion

type Err 

type Err int

Errors 

const (
	ErrSuccess Err = iota
	ErrNotFound
	ErrBadParameter
	ErrNotImplemented
	ErrConflict
	ErrInternalServerError
)

func (Err) Error 

func (e Err) Error() string

func (Err) With 

func (e Err) With(args ...interface{}) error

func (Err) Withf 

func (e Err) Withf(format string, args ...interface{}) error

type ImageMeta added in v0.0.10 

type ImageMeta struct {
	Url    string `json:"url,omitempty"`
	Data   []byte `json:"b64_json,omitempty"`
	Prompt string `json:"revised_prompt,omitempty"`
}

OpenAI image metadata

type Model 

type Model interface {
	// Return the name of the model
	Name() string

	// Return the description of the model
	Description() string

	// Return any model aliases
	Aliases() []string

	// Return am empty session context object for the model, setting
	// session options
	Context(...Opt) Context

	// Create a completion from a text prompt, including image
	// and audio (TTS) generation
	Completion(context.Context, string, ...Opt) (Completion, error)

	// Embedding vector generation
	Embedding(context.Context, string, ...Opt) ([]float64, error)
}

An Model can be used to generate a response to a user prompt, which is passed to an agent. A back-and-forth interaction occurs through a session context object.

type Opt 

type Opt func(*Opts) error

A generic option type, which can set options on an agent or session

func WithAgent 

func WithAgent(agent Agent) Opt

Set agent

func WithAttachment 

func WithAttachment(r io.Reader) Opt

Create an attachment

func WithFormat added in v0.0.3 

func WithFormat(v any) Opt

Set format for output, which is dependent on the model used

func WithFrequencyPenalty added in v0.0.3 

func WithFrequencyPenalty(v float64) Opt

func WithMaxTokens added in v0.0.3 

func WithMaxTokens(v uint64) Opt

The maximum number of tokens to generate in the completion.

func WithNumCompletions added in v0.0.3 

func WithNumCompletions(v uint64) Opt

Number of completions to return for each request

func WithPrediction added in v0.0.6 

func WithPrediction(v string) Opt

Predicted output, which is most common when you are regenerating a file with only minor changes to most of the content.

func WithPresencePenalty added in v0.0.3 

func WithPresencePenalty(v float64) Opt

func WithQuality added in v0.0.10 

func WithQuality(v string) Opt

Set quality for output (DALL-E)

func WithSafePrompt added in v0.0.3 

func WithSafePrompt() Opt

Inject a safety prompt before all conversations.

func WithSeed added in v0.0.3 

func WithSeed(v uint64) Opt

Set random seed for deterministic behavior

func WithSize added in v0.0.10 

func WithSize(v string) Opt

Set size for output (DALL-E)

func WithStopSequence added in v0.0.3 

func WithStopSequence(v ...string) Opt

Set stop sequence

func WithStream 

func WithStream(fn func(Completion)) Opt

Set chat streaming function

func WithStyle added in v0.0.10 

func WithStyle(v string) Opt

Set style for output (DALL-E)

func WithSystemPrompt 

func WithSystemPrompt(v string) Opt

Set system prompt

func WithTemperature 

func WithTemperature(v float64) Opt

The temperature of the model. Increasing the temperature will make the model answer more creatively.

func WithToolChoice added in v0.0.3 

func WithToolChoice(v ...string) Opt

Set tool choices: can be auto, none, required, any or a list of tool names

func WithToolKit 

func WithToolKit(toolkit ToolKit) Opt

Set toolkit of tools

func WithTopK 

func WithTopK(v uint64) Opt

Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative.

func WithTopP 

func WithTopP(v float64) Opt

Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text.

func WithUser added in v0.0.6 

func WithUser(v string) Opt

A unique identifier representing your end-user

type Opts 

type Opts struct {
	// contains filtered or unexported fields
}

set of options

func ApplyOpts 

func ApplyOpts(opts ...Opt) (*Opts, error)

ApplyOpts returns a structure of options

func ApplyPromptOpts added in v0.0.3 

func ApplyPromptOpts(opts ...Opt) (*Opts, error)

ApplyPromptOpts returns a structure of options for a prompt

func (*Opts) Agents 

func (o *Opts) Agents() []Agent

Return the array of registered agents

func (*Opts) Attachments 

func (o *Opts) Attachments() []*Attachment

Return attachments

func (*Opts) Get 

func (o *Opts) Get(key string) any

Get an option value

func (*Opts) GetBool 

func (o *Opts) GetBool(key string) bool

Get an option value as a boolean

func (*Opts) GetDuration 

func (o *Opts) GetDuration(key string) time.Duration

Get an option value as a duration

func (*Opts) GetFloat64 

func (o *Opts) GetFloat64(key string) float64

Get an option value as a float64

func (*Opts) GetString 

func (o *Opts) GetString(key string) string

Get an option value as a string

func (*Opts) GetUint64 

func (o *Opts) GetUint64(key string) uint64

Get an option value as an unsigned integer

func (*Opts) Has 

func (o *Opts) Has(key string) bool

Has an option value

func (Opts) MarshalJSON added in v0.0.3 

func (o Opts) MarshalJSON() ([]byte, error)

func (*Opts) Set 

func (o *Opts) Set(key string, value any)

Set an option value

func (*Opts) StreamFn 

func (o *Opts) StreamFn() func(Completion)

Return the stream function

func (Opts) String added in v0.0.3 

func (o Opts) String() string

func (*Opts) SystemPrompt 

func (o *Opts) SystemPrompt() string

Return the system prompt

func (*Opts) ToolKit 

func (o *Opts) ToolKit() ToolKit

Return the set of tools

type Tool 

type Tool interface {
	// The name of the tool
	Name() string

	// The description of the tool
	Description() string

	// Run the tool with a deadline and return the result
	Run(context.Context) (any, error)
}

Definition of a tool

type ToolCall 

type ToolCall interface {
	// The tool name
	Name() string

	// The tool identifier
	Id() string

	// Decode the calling parameters
	Decode(v any) error
}

A call-out to a tool

type ToolKit 

type ToolKit interface {
	// Register a tool in the toolkit
	Register(Tool) error

	// Return all the tools
	Tools(Agent) []Tool

	// Run the tool calls in parallel and return the results
	Run(context.Context, ...ToolCall) ([]ToolResult, error)
}

ToolKit is a collection of tools

type ToolResult 

type ToolResult interface {
	// The call associated with the result
	Call() ToolCall

	// The result, which can be encoded into json
	Value() any
}

Results from calling tools

Source Files

View all Source files

Directories

Path Synopsis
cmd
llm
examples
agents
completion
embedding
image_caption
image_generate
models
tool_adder
pkg
agent
anthropic
anthropic implements an API client for anthropic https://docs.anthropic.com/en/api/getting-started
 anthropic implements an API client for anthropic https://docs.anthropic.com/en/api/getting-started
gemini
gemini implements an API client for Google's Gemini LLM (https://ai.google.dev/gemini-api/docs)
 gemini implements an API client for Google's Gemini LLM (https://ai.google.dev/gemini-api/docs)
internal/impl
mistral
mistral implements an API client for mistral https://docs.mistral.ai/api/
 mistral implements an API client for mistral https://docs.mistral.ai/api/
newsapi
newsapi implements an API client for NewsAPI (https://newsapi.org/docs)
 newsapi implements an API client for NewsAPI (https://newsapi.org/docs)
ollama
ollama implements an API client for ollama https://github.com/ollama/ollama/blob/main/docs/api.md
 ollama implements an API client for ollama https://github.com/ollama/ollama/blob/main/docs/api.md
openai
openai implements an API client for OpenAI https://platform.openai.com/docs/api-reference
 openai implements an API client for OpenAI https://platform.openai.com/docs/api-reference
tool
ui
ui/telegram
ui/term
version

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.