llm

package module
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 2, 2025 License: Apache-2.0 Imports: 10 Imported by: 0

README

go-llm

Large Language Model API interface. This is a simple API interface for large language models which run on Ollama, Anthopic and Mistral (OpenAI might be added later).

The module includes the ability to utilize:

  • Maintaining a session of messages
  • Tool calling support, including using your own tools (aka Tool plugins)
  • Creating embedding vectors from text
  • Streaming responses
  • Multi-modal support (aka, Images and Attachments)

There is a command-line tool included in the module which can be used to interact with the API. If you have docker installed, you can use the following command to run the tool, without installation:

# Display help
docker run ghcr.io/mutablelogic/go-llm:latest --help

# Interact with Claude to retrieve news headlines, assuming
# you have an API key for Anthropic and NewsAPI
docker run \
  -e OLLAMA_URL -e MISTRAL_API_KEY -e NEWSAPI_KEY \
  ghcr.io/mutablelogic/go-llm:latest \
  chat mistral-small-latest --prompt "What is the latest news?" --no-stream

See below for more information on how to use the command-line tool (or how to install it if you have a go compiler).

Programmatic Usage

See the documentation here for integration into your own Go programs.

Agent Instantiation

For each LLM provider, you create an agent which can be used to interact with the API. To create an Ollama agent,

import (
  "github.com/mutablelogic/go-llm/pkg/ollama"
)

func main() {
  // Create a new agent - replace the URL with the one to your Ollama instance
  agent, err := ollama.New("https://ollama.com/api/v1/")
  if err != nil {
    panic(err)
  }
  // ...
}

To create an Anthropic agent with an API key stored as an environment variable,

import (
  "github.com/mutablelogic/go-llm/pkg/anthropic"
)

func main() {
  // Create a new agent
  agent, err := anthropic.New(os.Getenv("ANTHROPIC_API_KEY"))
  if err != nil {
    panic(err)
  }
  // ...
}

For Mistral models, you can use:

import (
  "github.com/mutablelogic/go-llm/pkg/mistral"
)

func main() {
  // Create a new agent
  agent, err := mistral.New(os.Getenv("MISTRAL_API_KEY"))
  if err != nil {
    panic(err)
  }
  // ...
}

You can append options to the agent creation to set the client/server communication options, such as user agent strings, timeouts, debugging, rate limiting, adding custom headers, etc. See here for more information.

There is also an aggregated agent which can be used to interact with multiple providers at once. This is useful if you want to use models from different providers simultaneously.

import (
  "github.com/mutablelogic/go-llm/pkg/agent"
)

func main() {
  // Create a new agent which aggregates multiple providers
  agent, err := agent.New(
    agent.WithAnthropic(os.Getenv("ANTHROPIC_API_KEY")), 
    agent.WithMistral(os.Getenv("MISTRAL_API_KEY")),
    agent.WithOllama(os.Getenv("OLLAMA_URL")),
  )
  if err != nil {
    panic(err)
  }
  // ...
}
Chat Sessions

You create a chat session with a model as follows,

import (
  "github.com/mutablelogic/go-llm"
)

func session(ctx context.Context, agent llm.Agent) error {
  // Create a new chat session
  session := agent.Model(context.TODO(), "claude-3-5-haiku-20241022").Context()

  // Repeat forever
  for {
    err := session.FromUser(ctx, "hello")
    if err != nil {
      return err
    }

    // Print the response for the zero'th completion
    fmt.Println(session.Text(0))
  }
}

The Context object will continue to store the current session and options, and will ensure the session is maintained across multiple calls.

Embedding Generation

TODO

Attachments & Image Caption Generation

TODO

Streaming

TODO

Tool Support

TODO

Options

You can add options to sessions, or to prompts. Different providers and models support different options.

type Model interface {
  // Set session-wide options
  Context(...Opt) Context

  // Add attachments (images, PDF's) to a user prompt for completion
  UserPrompt(string, ...Opt) Context

  // Create an embedding vector with embedding options
  Embedding(context.Context, string, ...Opt) ([]float64, error)
}

type Context interface {
  // Add single-use options when calling the model, which override
  // session options. You can attach files to a user prompt.
  FromUser(context.Context, string, ...Opt) error
}

The options are as follows:

Option Ollama Anthropic Mistral OpenAI Description
llm.WithTemperature(float64) Yes Yes Yes - What sampling temperature to use, between 0.0 and 1.0. Higher values like 0.7 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.
llm.WithTopP(float64) Yes Yes Yes - Nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.
llm.WithTopK(uint64) Yes Yes No - Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative.
llm.WithMaxTokens(uint64) No Yes Yes - The maximum number of tokens to generate in the response.
llm.WithStream(func(llm.Completion)) Can be enabled when tools are not used Yes Yes - Stream the response to a function.
llm.WithToolChoice(string, string, ...) No Yes Use auto, any, none, required or a function name. Only the first argument is used. - The tool to use for the model.
llm.WithToolKit(llm.ToolKit) Cannot be combined with streaming Yes Yes - The set of tools to use.
llm.WithStopSequence(string, string, ...) Yes Yes Yes - Stop generation if one of these tokens is detected.
llm.WithSystemPrompt(string) No Yes Yes - Set the system prompt for the model.
llm.WithSeed(uint64) Yes Yes Yes - The seed to use for random sampling. If set, different calls will generate deterministic results.
llm.WithFormat(string) Use json Yes Use json_format or text - The format of the response. For Mistral, you must also instruct the model to produce JSON yourself with a system or a user message.
llm.WithPresencePenalty(float64) Yes No Yes - Determines how much the model penalizes the repetition of words or phrases. A higher presence penalty encourages the model to use a wider variety of words and phrases, making the output more diverse and creative.
llm.WithFequencyPenalty(float64) Yes No Yes - Penalizes the repetition of words based on their frequency in the generated text. A higher frequency penalty discourages the model from repeating words that have already appeared frequently in the output, promoting diversity and reducing repetition.
mistral.WithPrediction(string) No No Yes - Enable users to specify expected results, optimizing response times by leveraging known or predictable content. This approach is especially effective for updating text documents or code files with minimal changes, reducing latency while maintaining high-quality results.
llm.WithSafePrompt() No No Yes - Whether to inject a safety prompt before all conversations.
llm.WithNumCompletions(uint64) No No Yes - Number of completions to return for each request.
llm.WithAttachment(io.Reader) Yes Yes Yes - Attach a file to a user prompt. It is the responsibility of the caller to close the reader.
antropic.WithEphemeral() No Yes No - Attachments should be cached server-side
antropic.WithCitations() No Yes No - Attachments should be used in citations
antropic.WithUser(string) No Yes No - Indicate the user name for the request, for debugging

The Command Line Tool

You can use the command-line tool to interact with the API. To build the tool, you can use the following command:

go install github.com/mutablelogic/go-llm/cmd/llm@latest
llm --help

The output is something like:

Usage: llm <command> [flags]

LLM agent command line interface

Flags:
  -h, --help                      Show context-sensitive help.
      --debug                     Enable debug output
      --verbose                   Enable verbose output
      --ollama-endpoint=STRING    Ollama endpoint ($OLLAMA_URL)
      --anthropic-key=STRING      Anthropic API Key ($ANTHROPIC_API_KEY)
      --news-key=STRING           News API Key ($NEWSAPI_KEY)

Commands:
  agents      Return a list of agents
  models      Return a list of models
  tools       Return a list of tools
  download    Download a model
  chat        Start a chat session

Run "llm <command> --help" for more information on a command.

Contributing & Distribution

This module is currently in development and subject to change. Please do file feature requests and bugs here. The license is Apache 2 so feel free to redistribute. Redistributions in either source code or binary form must reproduce the copyright notice, and please link back to this repository for more information:

go-llm
https://github.com/mutablelogic/go-llm/
Copyright (c) 2025 David Thorpe, All rights reserved.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Agent 

type Agent interface {
	// Return the name of the agent
	Name() string

	// Return the models
	Models(context.Context) ([]Model, error)

	// Return a model by name, or nil if not found.
	// Panics on error.
	Model(context.Context, string) Model
}

An LLM Agent is a client for the LLM service

type Attachment 

type Attachment struct {
	// contains filtered or unexported fields
}

Attachment for messages

func ReadAttachment 

func ReadAttachment(r io.Reader) (*Attachment, error)

ReadAttachment returns an attachment from a reader object. It is the responsibility of the caller to close the reader.

func (*Attachment) Data 

func (a *Attachment) Data() []byte

func (*Attachment) Filename 

func (a *Attachment) Filename() string

func (*Attachment) String added in v0.0.3 

func (a *Attachment) String() string

func (*Attachment) Type added in v0.0.3 

func (a *Attachment) Type() string

func (*Attachment) Url added in v0.0.3 

func (a *Attachment) Url() string

type Completion added in v0.0.3 

type Completion interface {
	// Return the number of completions, which is ususally 1 unless
	// WithNumCompletions was used when calling the model
	Num() int

	// Return the current session role, which can be system, assistant, user, tool, tool_result, ...
	// If this is a completion, the role is usually 'assistant'
	Role() string

	// Return the text for the last completion, with the argument as the
	// completion index (usually 0). If multiple completions are not
	// supported, the argument is ignored.
	Text(int) string

	// Return the current session tool calls given the completion index.
	// Will return nil if no tool calls were returned.
	ToolCalls(int) []ToolCall
}

Completion is the content of the last context message

type Context 

type Context interface {
	Completion

	// Generate a response from a user prompt (with attachments and
	// other options)
	FromUser(context.Context, string, ...Opt) error

	// Generate a response from a tool, passing the results
	// from the tool call
	FromTool(context.Context, ...ToolResult) error
}

Context is fed to the agent to generate a response

type Err 

type Err int

Errors 

const (
	ErrSuccess Err = iota
	ErrNotFound
	ErrBadParameter
	ErrNotImplemented
	ErrConflict
)

func (Err) Error 

func (e Err) Error() string

func (Err) With 

func (e Err) With(args ...interface{}) error

func (Err) Withf 

func (e Err) Withf(format string, args ...interface{}) error

type Model 

type Model interface {
	// Return the name of the model
	Name() string

	// Return am empty session context object for the model,
	// setting session options
	Context(...Opt) Context

	// Convenience method to create a session context object
	// with a user prompt
	UserPrompt(string, ...Opt) Context

	// Embedding vector generation
	Embedding(context.Context, string, ...Opt) ([]float64, error)
}

An Model can be used to generate a response to a user prompt, which is passed to an agent. The interaction occurs through a session context object.

type Opt 

type Opt func(*Opts) error

A generic option type, which can set options on an agent or session

func WithAgent 

func WithAgent(agent Agent) Opt

Set agent

func WithAttachment 

func WithAttachment(r io.Reader) Opt

Create an attachment

func WithFormat added in v0.0.3 

func WithFormat(v any) Opt

Set format

func WithFrequencyPenalty added in v0.0.3 

func WithFrequencyPenalty(v float64) Opt

func WithMaxTokens added in v0.0.3 

func WithMaxTokens(v uint64) Opt

The maximum number of tokens to generate in the completion.

func WithNumCompletions added in v0.0.3 

func WithNumCompletions(v uint64) Opt

Number of completions to return for each request

func WithPresencePenalty added in v0.0.3 

func WithPresencePenalty(v float64) Opt

func WithSafePrompt added in v0.0.3 

func WithSafePrompt() Opt

Inject a safety prompt before all conversations.

func WithSeed added in v0.0.3 

func WithSeed(v uint64) Opt

Set random seed for deterministic behavior

func WithStopSequence added in v0.0.3 

func WithStopSequence(v ...string) Opt

Set stop sequence

func WithStream 

func WithStream(fn func(Completion)) Opt

Set chat streaming function

func WithSystemPrompt 

func WithSystemPrompt(v string) Opt

Set system prompt

func WithTemperature 

func WithTemperature(v float64) Opt

The temperature of the model. Increasing the temperature will make the model answer more creatively.

func WithToolChoice added in v0.0.3 

func WithToolChoice(v ...string) Opt

Set tool choices: can be auto, none, required, any or a list of tool names

func WithToolKit 

func WithToolKit(toolkit ToolKit) Opt

Set toolkit of tools

func WithTopK 

func WithTopK(v uint64) Opt

Reduces the probability of generating nonsense. A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative.

func WithTopP 

func WithTopP(v float64) Opt

Works together with top-k. A higher value (e.g., 0.95) will lead to more diverse text, while a lower value (e.g., 0.5) will generate more focused and conservative text.

type Opts 

type Opts struct {
	// contains filtered or unexported fields
}

set of options

func ApplyOpts 

func ApplyOpts(opts ...Opt) (*Opts, error)

ApplyOpts returns a structure of options

func ApplyPromptOpts added in v0.0.3 

func ApplyPromptOpts(opts ...Opt) (*Opts, error)

ApplyPromptOpts returns a structure of options for a prompt

func (*Opts) Agents 

func (o *Opts) Agents() []Agent

Return the array of registered agents

func (*Opts) Attachments 

func (o *Opts) Attachments() []*Attachment

Return attachments

func (*Opts) Get 

func (o *Opts) Get(key string) any

Get an option value

func (*Opts) GetBool 

func (o *Opts) GetBool(key string) bool

Get an option value as a boolean

func (*Opts) GetDuration 

func (o *Opts) GetDuration(key string) time.Duration

Get an option value as a duration

func (*Opts) GetFloat64 

func (o *Opts) GetFloat64(key string) float64

Get an option value as a float64

func (*Opts) GetString 

func (o *Opts) GetString(key string) string

Get an option value as a string

func (*Opts) GetUint64 

func (o *Opts) GetUint64(key string) uint64

Get an option value as an unsigned integer

func (*Opts) Has 

func (o *Opts) Has(key string) bool

Has an option value

func (Opts) MarshalJSON added in v0.0.3 

func (o Opts) MarshalJSON() ([]byte, error)

func (*Opts) Set 

func (o *Opts) Set(key string, value any)

Set an option value

func (*Opts) StreamFn 

func (o *Opts) StreamFn() func(Completion)

Return the stream function

func (Opts) String added in v0.0.3 

func (o Opts) String() string

func (*Opts) SystemPrompt 

func (o *Opts) SystemPrompt() string

Return the system prompt

func (*Opts) ToolKit 

func (o *Opts) ToolKit() ToolKit

Return the set of tools

type Tool 

type Tool interface {
	// The name of the tool
	Name() string

	// The description of the tool
	Description() string

	// Run the tool with a deadline and return the result
	Run(context.Context) (any, error)
}

Definition of a tool

type ToolCall 

type ToolCall interface {
	// The tool name
	Name() string

	// The tool identifier
	Id() string

	// Decode the calling parameters
	Decode(v any) error
}

A call-out to a tool

type ToolKit 

type ToolKit interface {
	// Register a tool in the toolkit
	Register(Tool) error

	// Return all the tools
	Tools(Agent) []Tool

	// Run the tool calls in parallel and return the results
	Run(context.Context, ...ToolCall) ([]ToolResult, error)
}

ToolKit is a collection of tools

type ToolResult 

type ToolResult interface {
	// The call associated with the result
	Call() ToolCall

	// The result, which can be encoded into json
	Value() any
}

Results from calling tools

Source Files

View all Source files

Directories

Path Synopsis
cmd
llm
pkg
agent
anthropic
anthropic implements an API client for anthropic (https://docs.anthropic.com/en/api/getting-started)
 anthropic implements an API client for anthropic (https://docs.anthropic.com/en/api/getting-started)
mistral
mistral implements an API client for mistral (https://docs.mistral.ai/api/)
 mistral implements an API client for mistral (https://docs.mistral.ai/api/)
newsapi
newsapi implements an API client for NewsAPI (https://newsapi.org/docs)
 newsapi implements an API client for NewsAPI (https://newsapi.org/docs)
ollama
ollama implements an API client for ollama https://github.com/ollama/ollama/blob/main/docs/api.md
 ollama implements an API client for ollama https://github.com/ollama/ollama/blob/main/docs/api.md
tool

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.