llmkit

README

llmkit

Website: https://richardwooding.github.io/llmkit/

One Go client for twelve AI back-ends. Pick a model by name, code against a one-method interface, and swap vendors without touching call sites.

chat, err := llmkit.Open[llmkit.Chatter]("claude-sonnet-4-5")
resp, err := chat.Chat(ctx, &llmkit.Request{
	Messages: []llmkit.Message{llmkit.UserText("Explain iter.Seq2 in one paragraph.")},
})
fmt.Println(resp.Text())

Pure Go 1.27, no cgo. The core module depends only on the standard library and golang.org/x/oauth2 (Google credentials for Vertex AI). Vertex AI over gRPC lives in a separate nested module so its dependency tree stays opt-in.

Why

• Small interfaces. Chatter, Streamer, Embedder and Reranker each have one method. Ask for exactly what you need with llmkit.Open[T]; a provider that lacks the capability fails at construction with ErrUnsupported, before any network call.
• Model-name routing. "gpt-5", "claude-sonnet-4-5", "gemini-2.5-pro", "deepseek-reasoner", "grok-4", "command-a-03-2025", "voyage-3-large" and "llama3.2:3b" resolve on their own. Anything ambiguous takes a prefix: "groq/llama-3.3-70b-versatile", "openrouter/openai/gpt-4o", "hf/meta-llama/Llama-3.3-70B-Instruct", "vertexgrpc/gemini-2.5-flash". Bare open-weight names fall back to a local Ollama daemon.
• One message model. Text, images, audio, documents, reasoning, tool calls and tool results are typed parts; each provider maps what it supports and rejects the rest up front.
• Streaming as iterators. Stream returns iter.Seq2[Chunk, error]; the request is sent when you start ranging and closed when you stop.
• Escape hatches. Request.Extra and Request.ProviderOptions merge raw fields into the wire body; every response keeps its raw JSON.

Install

go get github.com/richardwooding/llmkit
go get github.com/richardwooding/llmkit/vertexgrpc              # optional, Vertex AI over gRPC
go install github.com/richardwooding/llmkit/cmd/llmkit@latest   # optional CLI

Providers

Provider	Names	Chat	Stream	Embed	Rerank	Tools	Image	Audio	File/PDF	Auth
OpenAI (Responses API)	gpt-*, o*, text-embedding-3-*	✅	✅	✅	–	✅	✅	–	✅	OPENAI_API_KEY
DeepSeek	deepseek-*	✅	✅	–	–	✅¹	–	–	–	DEEPSEEK_API_KEY
Ollama	name:tag, bare fallback	✅	✅	✅	–	✅	✅	–	–	OLLAMA_HOST
Vertex AI (REST)	gemini-*, text-embedding-*	✅	✅	✅	–	✅	✅	✅	✅	ADC / GOOGLE_CLOUD_PROJECT
Vertex AI (gRPC)	vertexgrpc/…	✅	✅	✅	–	✅	✅	✅	✅	ADC
Anthropic	claude-*	✅	✅	–	–	✅	✅	–	✅	ANTHROPIC_API_KEY
Cohere	command*, embed-*, rerank-*	✅	✅	✅	✅	✅	✅	–	–	COHERE_API_KEY
Groq	groq/…	✅	✅	–	–	✅	✅	–	–	GROQ_API_KEY
x.ai (Grok)	grok-*	✅	✅	–	–	✅	✅	–	–	XAI_API_KEY
Hugging Face	org/model, hf/…	✅	✅	✅	–	✅	✅	–	–	HF_TOKEN
OpenRouter	openrouter/…	✅	✅	✅	–	✅	✅	✅	✅	OPENROUTER_API_KEY
VoyageAI	voyage-*	–	–	✅²	✅	–	–	–	–	VOYAGE_API_KEY

¹ deepseek-reasoner rejects tool definitions; llmkit fails fast with ErrUnsupported. ² VoyageAI also implements MultimodalEmbedder for text + image + video inputs.

Every provider reads its key from the environment variable shown, or from llmkit.WithAPIKey. llmkit.WithBaseURL, WithHTTPClient, WithHeader and WithTimeout apply to all of them.

Usage

Streaming

stream, err := llmkit.Open[llmkit.Streamer]("llama3.2")
for chunk, err := range stream.Stream(ctx, req) {
	if err != nil {
		return err
	}
	switch chunk.Kind {
	case llmkit.ChunkText:
		fmt.Print(chunk.Text)
	case llmkit.ChunkFinish:
		fmt.Println("\n", chunk.Usage.OutputTokens, "tokens")
	}
}

llmkit.Collect(stream.Stream(ctx, req)) turns a stream back into a *Response.

Tool calling

req := &llmkit.Request{
	Messages: []llmkit.Message{llmkit.UserText("Weather in Cape Town?")},
	Tools: []llmkit.Tool{{
		Name:        "weather",
		Description: "Current weather for a city",
		Parameters:  json.RawMessage(`{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}`),
	}},
}
tools := map[string]llmkit.ToolFunc{
	"weather": func(ctx context.Context, args json.RawMessage) (string, error) {
		var in struct{ City string `json:"city"` }
		if err := json.Unmarshal(args, &in); err != nil {
			return "", err
		}
		return lookup(in.City), nil
	},
}
resp, err := llmkit.RunTools(ctx, chat, req, tools, 5)

RunTools appends assistant and tool messages to req.Messages until the model stops calling tools, feeding tool errors back as error results.

Multimodal input

png, _ := os.ReadFile("chart.png")
pdf, _ := os.ReadFile("report.pdf")
req := &llmkit.Request{Messages: []llmkit.Message{llmkit.User(
	llmkit.Text("Summarise the chart and the report."),
	llmkit.Image(png, "image/png"),
	llmkit.File(pdf, "application/pdf", "report.pdf"),
)}}

Providers that cannot accept a part return ErrUnsupported before sending anything.

Embeddings

embed, err := llmkit.Open[llmkit.Embedder]("voyage-3-large")
out, err := embed.Embed(ctx, &llmkit.EmbedRequest{
	Inputs:    []string{"first document", "second document"},
	InputType: llmkit.EmbedDocument,
})
vectors := out.Embeddings // [][]float32, one per input

Rerank

rr, err := llmkit.Open[llmkit.Reranker]("rerank-v3.5")
out, err := rr.Rerank(ctx, &llmkit.RerankRequest{Query: "go iterators", Documents: docs, TopN: 3})
for _, r := range out.Results { // best first
	fmt.Println(docs[r.Index], r.Score)
}

Command line

llmkit chat -m claude-sonnet-4-5 "Explain iter.Seq2 in one paragraph"
echo "Summarise this" | llmkit chat -m llama3.2 -stream
llmkit embed -m voyage-3-large "first" "second"
llmkit resolve gpt-5 openrouter/openai/gpt-4o meta-llama/Llama-3.3-70B-Instruct

Custom OpenAI-compatible endpoints

llmkit.Register(openaicompat.NewProvider(openaicompat.Config{
	ID:          "vllm",
	BaseURL:     "http://gpu-box:8000/v1",
	KeyOptional: true,
	Quirks:      openaicompat.Quirks{Images: true, StreamUsage: true},
}))
chat, err := llmkit.Open[llmkit.Chatter]("vllm/my-finetune")

Vertex AI over gRPC

import "github.com/richardwooding/llmkit/vertexgrpc"

llmkit.Register(vertexgrpc.Provider{})
chat, err := llmkit.Open[llmkit.Chatter]("vertexgrpc/gemini-2.5-flash",
	vertexgrpc.WithProject("my-project"), vertexgrpc.WithLocation("europe-west1"))

Errors, retries and rate limits

Provider failures are *llmkit.APIError values carrying the HTTP status, the provider's error code and any Retry-After hint. errors.Is(err, llmkit.ErrRateLimited) and errors.Is(err, llmkit.ErrContextLength) work across vendors. llmkit does not retry; compose an http.RoundTripper such as hostrate via WithHTTPClient.

Resolution rules

1. If the text before the first / is a registered provider ID or alias, that provider gets the rest (openrouter/openai/gpt-4o).
2. Otherwise the first provider whose Matches accepts the bare name wins: OpenAI, Anthropic, Vertex, DeepSeek, x.ai, Cohere, VoyageAI prefixes; Ollama for anything containing :; Hugging Face for org/model.
3. Otherwise the fallback: llmkit.SetFallback, then LLMKIT_DEFAULT_PROVIDER, then ollama.

What this is not

• Not an agent framework. RunTools is a small loop; bring your own planning, memory and observability.
• Not a retry or caching layer. Both belong in the http.Client you pass in.
• Not a wrapper around vendor SDKs. Every REST provider is written against the wire format with net/http; only vertexgrpc pulls in Google's client.

Changelog

See CHANGELOG.md.

License

MIT © 2026 Richard Wooding

Documentation

Overview

Package llmkit is a multi-vendor AI client. One factory resolves a model name such as "gpt-5", "claude-sonnet-4-5", "llama3.2:3b" or "openrouter/openai/gpt-4o" to a provider, and callers program against small single-method interfaces:

chat, err := llmkit.Open[llmkit.Chatter]("gpt-5")
resp, err := chat.Chat(ctx, &llmkit.Request{Messages: []llmkit.Message{llmkit.UserText("hi")}})

Every type in this package is an alias of the same name in package core, so provider packages and applications share one set of types.

Example

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"

	"github.com/richardwooding/llmkit"
	"github.com/richardwooding/llmkit/openaicompat"
)

func fakeServer() *httptest.Server {
	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var req struct {
			Stream   bool `json:"stream"`
			Messages []struct {
				Role string `json:"role"`
			} `json:"messages"`
		}
		_ = json.NewDecoder(r.Body).Decode(&req)
		switch {
		case req.Stream:
			w.Header().Set("Content-Type", "text/event-stream")
			_, _ = io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"Hello\"}}]}\n\n"+
				"data: {\"choices\":[{\"delta\":{\"content\":\", world\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n")
		case len(req.Messages) == 1:
			_, _ = io.WriteString(w, `{"choices":[{"message":{"role":"assistant","tool_calls":[{"id":"c1","type":"function","function":{"name":"weather","arguments":"{\"city\":\"Cape Town\"}"}}]},"finish_reason":"tool_calls"}]}`)
		default:
			_, _ = io.WriteString(w, `{"choices":[{"message":{"role":"assistant","content":"It is 24°C in Cape Town."},"finish_reason":"stop"}]}`)
		}
	}))
}

func main() {
	srv := fakeServer()
	defer srv.Close()
	// Any OpenAI-compatible server can be registered under its own prefix.
	llmkit.Register(openaicompat.NewProvider(openaicompat.Config{ID: "demo", BaseURL: srv.URL, KeyOptional: true}))

	chat, err := llmkit.Open[llmkit.Chatter]("demo/my-model")
	if err != nil {
		panic(err)
	}
	req := &llmkit.Request{
		Messages: []llmkit.Message{llmkit.UserText("What's the weather in Cape Town?")},
		Tools: []llmkit.Tool{{
			Name:       "weather",
			Parameters: json.RawMessage(`{"type":"object","properties":{"city":{"type":"string"}}}`),
		}},
	}
	tools := map[string]llmkit.ToolFunc{
		"weather": func(_ context.Context, args json.RawMessage) (string, error) {
			var in struct{ City string }
			_ = json.Unmarshal(args, &in)
			return "24°C in " + in.City, nil
		},
	}
	resp, err := llmkit.RunTools(context.Background(), chat, req, tools, 5)
	if err != nil {
		panic(err)
	}
	fmt.Println(resp.Text())

	stream, _ := llmkit.Open[llmkit.Streamer]("demo/my-model")
	for chunk, err := range stream.Stream(context.Background(), &llmkit.Request{Messages: []llmkit.Message{llmkit.UserText("Say hello")}}) {
		if err != nil {
			panic(err)
		}
		if chunk.Kind == llmkit.ChunkText {
			fmt.Print(chunk.Text)
		}
	}
	fmt.Println()
}

Output:
It is 24°C in Cape Town.
Hello, world

Index

• Constants
• Variables
• func As[T any](c core.Client) (T, error)
• func Collect(seq iter.Seq2[core.Chunk, error]) (*core.Response, error)
• func New(model string, opts ...core.Option) (core.Client, error)
• func Open[T any](model string, opts ...core.Option) (T, error)
• func OpenWith[T any](r *Registry, model string, opts ...core.Option) (T, error)
• func ParseModel(s string) (core.Provider, string, error)
• func Register(p core.Provider, aliases ...string)
• func RunTools(ctx context.Context, c core.Chatter, req *core.Request, ...) (*core.Response, error)
• func SetFallback(id string)
• type APIError
• type AudioPart
• type Chatter
• type Chunk
• type ChunkKind
• type Client
• type Config
• type EmbedInputType
• type EmbedRequest
• type EmbedResponse
• type Embedder
• type FilePart
• type FinishReason
• type ImagePart
• type Message
• type MultimodalEmbedRequest
• type MultimodalEmbedder
• type Option
• type Part
• type Provider
• type ReasoningConfig
• type ReasoningPart
• type Registry
  • func NewRegistry(providers ...core.Provider) *Registry
  • func (r *Registry) Lookup(id string) (core.Provider, bool)
  • func (r *Registry) New(model string, opts ...core.Option) (core.Client, error)
  • func (r *Registry) ParseModel(s string) (core.Provider, string, error)
  • func (r *Registry) Providers() []core.Provider
  • func (r *Registry) Register(p core.Provider, aliases ...string)
  • func (r *Registry) SetFallback(id string)
• type Request
• type RerankRequest
• type RerankResponse
• type RerankResult
• type Reranker
• type Response
• type ResponseFormat
• type Role
• type Streamer
• type TextPart
• type Tool
• type ToolCall
• type ToolCallDelta
• type ToolChoice
• type ToolChoiceMode
• type ToolFunc
• type ToolResult
• type Usage

Examples

• Package
• ParseModel

Constants

const (
	RoleSystem    = core.RoleSystem
	RoleUser      = core.RoleUser
	RoleAssistant = core.RoleAssistant
	RoleTool      = core.RoleTool
)

Roles.

const (
	ToolChoiceAuto     = core.ToolChoiceAuto
	ToolChoiceNone     = core.ToolChoiceNone
	ToolChoiceRequired = core.ToolChoiceRequired
	ToolChoiceNamed    = core.ToolChoiceNamed
)

Tool choice modes.

const (
	FormatJSON       = core.FormatJSON
	FormatJSONSchema = core.FormatJSONSchema
)

Response formats.

const (
	FinishStop          = core.FinishStop
	FinishLength        = core.FinishLength
	FinishToolCalls     = core.FinishToolCalls
	FinishContentFilter = core.FinishContentFilter
	FinishOther         = core.FinishOther
)

Finish reasons.

const (
	ChunkText      = core.ChunkText
	ChunkReasoning = core.ChunkReasoning
	ChunkToolCall  = core.ChunkToolCall
	ChunkFinish    = core.ChunkFinish
)

Chunk kinds.

const (
	EmbedQuery    = core.EmbedQuery
	EmbedDocument = core.EmbedDocument
)

Embedding input types.

const DefaultProviderEnv = "LLMKIT_DEFAULT_PROVIDER"

DefaultProviderEnv names the environment variable consulted for the fallback provider when a bare model name matches nothing.

Variables

var (
	ErrUnsupported      = core.ErrUnsupported
	ErrUnknownProvider  = core.ErrUnknownProvider
	ErrMissingAPIKey    = core.ErrMissingAPIKey
	ErrRateLimited      = core.ErrRateLimited
	ErrContextLength    = core.ErrContextLength
	ErrToolLoopExceeded = core.ErrToolLoopExceeded
)

Sentinel errors.

var (
	Text           = core.Text
	Image          = core.Image
	ImageURL       = core.ImageURL
	Audio          = core.Audio
	File           = core.File
	FileURL        = core.FileURL
	ToolResultText = core.ToolResultText
	System         = core.System
	User           = core.User
	UserText       = core.UserText
	Assistant      = core.Assistant
	ToolResults    = core.ToolResults
)

Message constructors.

var (
	WithAPIKey     = core.WithAPIKey
	WithBaseURL    = core.WithBaseURL
	WithHTTPClient = core.WithHTTPClient
	WithHeader     = core.WithHeader
	WithTimeout    = core.WithTimeout
	WithValue      = core.WithValue
	NewConfig      = core.NewConfig
)

Options.

var Default = newDefault()

Default is the registry used by the package-level functions. Providers are listed in bare-name match priority: strict prefixes first, then Ollama (tagged names and the fallback), then Hugging Face ("org/model" names).

Functions

func As

func As[T any](c core.Client) (T, error)

As asserts c to T, returning ErrUnsupported when the provider lacks the capability.

func Collect

func Collect(seq iter.Seq2[core.Chunk, error]) (*core.Response, error)

Collect drains a stream into a Response, concatenating text and reasoning deltas and reassembling tool-call arguments by index.

func New

func New(model string, opts ...core.Option) (core.Client, error)

New opens a client for model using the Default registry.

func Open

func Open[T any](model string, opts ...core.Option) (T, error)

Open resolves model and asserts the client to T, typically Chatter, Streamer, Embedder or an interface combining them.

func OpenWith

func OpenWith[T any](r *Registry, model string, opts ...core.Option) (T, error)

OpenWith is Open against a specific Registry.

func ParseModel

func ParseModel(s string) (core.Provider, string, error)

ParseModel resolves a model name against the Default registry.

Example

package main

import (
	"fmt"

	"github.com/richardwooding/llmkit"
)

func main() {
	for _, name := range []string{"gpt-5", "claude-sonnet-4-5", "llama3.2:3b", "openrouter/openai/gpt-4o", "meta-llama/Llama-3.3-70B-Instruct"} {
		p, model, err := llmkit.ParseModel(name)
		if err != nil {
			fmt.Println(name, "→", err)
			continue
		}
		fmt.Printf("%-36s → %s / %s\n", name, p.ID(), model)
	}
}

Output:
gpt-5                                → openai / gpt-5
claude-sonnet-4-5                    → anthropic / claude-sonnet-4-5
llama3.2:3b                          → ollama / llama3.2:3b
openrouter/openai/gpt-4o             → openrouter / openai/gpt-4o
meta-llama/Llama-3.3-70B-Instruct    → huggingface / meta-llama/Llama-3.3-70B-Instruct

func Register

func Register(p core.Provider, aliases ...string)

Register adds a provider to the Default registry.

func RunTools

func RunTools(ctx context.Context, c core.Chatter, req *core.Request, tools map[string]ToolFunc, maxIter int) (*core.Response, error)

RunTools chats until the model stops requesting tools or maxIter calls have been made, appending assistant and tool messages to req.Messages as it goes. Tool errors and unknown tool names are fed back to the model as error results; context errors abort. Usage across iterations is summed.

func SetFallback

func SetFallback(id string)

SetFallback sets the Default registry's fallback provider.

Types

type APIError

type APIError = core.APIError

APIError is an alias of core.APIError.

type AudioPart

type AudioPart = core.AudioPart

AudioPart is an alias of core.AudioPart.

type Chatter

type Chatter = core.Chatter

Chatter is an alias of core.Chatter.

type Chunk

type Chunk = core.Chunk

Chunk is an alias of core.Chunk.

type ChunkKind

type ChunkKind = core.ChunkKind

ChunkKind is an alias of core.ChunkKind.

type Client

type Client = core.Client

Client is an alias of core.Client.

type Config

type Config = core.Config

Config is an alias of core.Config.

type EmbedInputType

type EmbedInputType = core.EmbedInputType

EmbedInputType is an alias of core.EmbedInputType.

type EmbedRequest

type EmbedRequest = core.EmbedRequest

EmbedRequest is an alias of core.EmbedRequest.

type EmbedResponse

type EmbedResponse = core.EmbedResponse

EmbedResponse is an alias of core.EmbedResponse.

type Embedder

type Embedder = core.Embedder

Embedder is an alias of core.Embedder.

type FilePart

type FilePart = core.FilePart

FilePart is an alias of core.FilePart.

type FinishReason

type FinishReason = core.FinishReason

FinishReason is an alias of core.FinishReason.

type ImagePart

type ImagePart = core.ImagePart

ImagePart is an alias of core.ImagePart.

type Message

type Message = core.Message

Message is an alias of core.Message.

type MultimodalEmbedRequest

type MultimodalEmbedRequest = core.MultimodalEmbedRequest

MultimodalEmbedRequest is an alias of core.MultimodalEmbedRequest.

type MultimodalEmbedder

type MultimodalEmbedder = core.MultimodalEmbedder

MultimodalEmbedder is an alias of core.MultimodalEmbedder.

type Option

type Option = core.Option

Option is an alias of core.Option.

type Part

type Part = core.Part

Part is an alias of core.Part.

type Provider

type Provider = core.Provider

Provider is an alias of core.Provider.

type ReasoningConfig

type ReasoningConfig = core.ReasoningConfig

ReasoningConfig is an alias of core.ReasoningConfig.

type ReasoningPart

type ReasoningPart = core.ReasoningPart

ReasoningPart is an alias of core.ReasoningPart.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry resolves model names to providers. Registration order is match priority for bare names.

func NewRegistry

func NewRegistry(providers ...core.Provider) *Registry

NewRegistry builds a Registry holding providers in the given order.

func (*Registry) Lookup

func (r *Registry) Lookup(id string) (core.Provider, bool)

Lookup returns the provider registered under id or an alias.

func (*Registry) New

func (r *Registry) New(model string, opts ...core.Option) (core.Client, error)

New resolves model and opens a client for it.

func (*Registry) ParseModel

func (r *Registry) ParseModel(s string) (core.Provider, string, error)

ParseModel splits "<provider>/<model>" or resolves a bare "<model>" to the provider that claims it, then the fallback.

func (*Registry) Providers

func (r *Registry) Providers() []core.Provider

Providers returns the registered providers in priority order.

func (*Registry) Register

func (r *Registry) Register(p core.Provider, aliases ...string)

Register adds a provider under its ID and any aliases, replacing an earlier provider with the same ID.

func (*Registry) SetFallback

func (r *Registry) SetFallback(id string)

SetFallback names the provider used for bare model names nothing claims. It takes precedence over LLMKIT_DEFAULT_PROVIDER.

type Request

type Request = core.Request

Request is an alias of core.Request.

type RerankRequest

type RerankRequest = core.RerankRequest

RerankRequest is an alias of core.RerankRequest.

type RerankResponse

type RerankResponse = core.RerankResponse

RerankResponse is an alias of core.RerankResponse.

type RerankResult

type RerankResult = core.RerankResult

RerankResult is an alias of core.RerankResult.

type Reranker

type Reranker = core.Reranker

Reranker is an alias of core.Reranker.

type Response

type Response = core.Response

Response is an alias of core.Response.

type ResponseFormat

type ResponseFormat = core.ResponseFormat

ResponseFormat is an alias of core.ResponseFormat.

type Role

type Role = core.Role

Role is an alias of core.Role.

type Streamer

type Streamer = core.Streamer

Streamer is an alias of core.Streamer.

type TextPart

type TextPart = core.TextPart

TextPart is an alias of core.TextPart.

type Tool

type Tool = core.Tool

Tool is an alias of core.Tool.

type ToolCall

type ToolCall = core.ToolCall

ToolCall is an alias of core.ToolCall.

type ToolCallDelta

type ToolCallDelta = core.ToolCallDelta

ToolCallDelta is an alias of core.ToolCallDelta.

type ToolChoice

type ToolChoice = core.ToolChoice

ToolChoice is an alias of core.ToolChoice.

type ToolChoiceMode

type ToolChoiceMode = core.ToolChoiceMode

ToolChoiceMode is an alias of core.ToolChoiceMode.

type ToolFunc

type ToolFunc func(ctx context.Context, args json.RawMessage) (string, error)

ToolFunc executes one tool call and returns its textual result.

type ToolResult

type ToolResult = core.ToolResult

ToolResult is an alias of core.ToolResult.

type Usage

type Usage = core.Usage

Usage is an alias of core.Usage.