llm

package module
v0.33.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 8, 2026 License: MIT Imports: 21 Imported by: 0

README

LLM Provider Abstraction Library

A unified Go library for interacting with multiple LLM providers through a consistent interface. Supports streaming responses, tool calling, reasoning, prompt caching, and zero-config multi-provider setup.

Features

  • Unified Provider Interface — Single API for multiple LLM providers
  • Streaming Support — Channel-based streaming with structured delta events
  • Tool Calling — Consistent tool/function calling across providers
  • Typed Tool Dispatch — StreamResponse handles tool calls with strongly-typed handlers
  • Reasoning Support — Extended thinking / reasoning tokens (Anthropic, OpenAI o-series, Bedrock)
  • Prompt Caching — Transparent cache control for Anthropic, Bedrock, and OpenAI
  • Context Cancellation — Proper cancellation support for long-running streams
  • Zero-config Setup — provider/auto auto-detects providers from environment variables
  • llmtest Package — Test helpers for stream consumers (net/http/httptest style)

Supported Providers

Provider Name Description
Anthropic API anthropic Direct Anthropic API with API key
Claude OAuth claude OAuth-based Claude access (auto-detects local credentials)
OpenAI openai OpenAI GPT models (GPT-4o, GPT-5, o-series, Codex)
AWS Bedrock bedrock AWS Bedrock models (Claude, Llama, etc.)
MiniMax minimax MiniMax M2 models via Anthropic-compatible API
Ollama ollama Local Ollama models
OpenRouter openrouter 200+ tool-enabled models via OpenRouter proxy
Router router Combines multiple providers with failover and aliases

Installation

go get github.com/codewandler/llm

Quick Start

package main

import (
    "context"
    "fmt"

    "github.com/codewandler/llm"
    "github.com/codewandler/llm/provider/auto"
)

func main() {
    ctx := context.Background()

    // Auto-detects providers from environment variables
    p, err := auto.New(ctx)
    if err != nil {
        panic(err)
    }

    events, err := p.CreateStream(ctx, llm.StreamRequest{
        Model: "anthropic/claude-sonnet-4-5",
        Messages: llm.Messages{
            &llm.UserMsg{Content: "What is the capital of France?"},
        },
    })
    if err != nil {
        panic(err)
    }

    for event := range events {
        switch event.Type {
        case llm.StreamEventDelta:
            fmt.Print(event.Text())
        case llm.StreamEventDone:
            fmt.Println()
            if event.Usage != nil {
                fmt.Printf("Tokens: %d in, %d out\n",
                    event.Usage.InputTokens, event.Usage.OutputTokens)
            }
        case llm.StreamEventError:
            fmt.Printf("Error: %v\n", event.Error)
        }
    }
}

Provider Setup

provider/auto — Zero-Config Multi-Provider

auto.New(ctx, ...Option) auto-detects providers from environment variables and returns a ready-to-use llm.Provider:

import "github.com/codewandler/llm/provider/auto"

// Auto-detect everything from environment variables
p, err := auto.New(ctx)

// Or explicitly opt in to specific providers
p, err := auto.New(ctx,
    auto.WithAnthropic(),     // ANTHROPIC_API_KEY
    auto.WithOpenAI(),        // OPENAI_KEY or OPENAI_API_KEY
    auto.WithBedrock(),       // AWS credentials
    auto.WithOpenRouter(),    // OPENROUTER_API_KEY
    auto.WithClaudeLocal(),   // ~/.claude/.credentials.json
)

// Add a Claude OAuth account from a token store
p, err := auto.New(ctx, auto.WithClaude(myTokenStore))

// Custom global aliases with failover
p, err := auto.New(ctx,
    auto.WithOpenAI(),
    auto.WithOpenRouter(),
    auto.WithGlobalAlias("o3", "openai/o3", "openrouter/openai/o3"),
)
Direct Provider Usage

Each provider can also be used directly without auto:

import "github.com/codewandler/llm/provider/anthropic"

p := anthropic.New(llm.APIKeyFromEnv("ANTHROPIC_API_KEY"))

events, err := p.CreateStream(ctx, llm.StreamRequest{
    Model:    "claude-sonnet-4-5",
    Messages: llm.Messages{&llm.UserMsg{Content: "Hello!"}},
})

import "github.com/codewandler/llm/provider/openai"

p := openai.New(llm.APIKeyFromEnv("OPENAI_KEY"))

import "github.com/codewandler/llm/provider/bedrock"

p := bedrock.New() // uses default AWS credential chain
p := bedrock.New(bedrock.WithRegion("us-east-1"))

import "github.com/codewandler/llm/provider/ollama"

p := ollama.New("http://localhost:11434")

import "github.com/codewandler/llm/provider/openrouter"

p := openrouter.New(llm.APIKeyFromEnv("OPENROUTER_API_KEY"))
Claude OAuth Provider
import "github.com/codewandler/llm/provider/anthropic/claude"

// Auto-detect local Claude credentials (default)
p := claude.New()

// Or with explicit token provider
p := claude.New(
    claude.WithManagedTokenProvider("my-key", tokenStore, nil),
)

Token management interfaces:

  • TokenStore — stores and retrieves tokens (implement for your storage backend)
  • LocalTokenStore — reads from ~/.claude/.credentials.json
  • ManagedTokenProvider — wraps a TokenStore with automatic refresh
Router Provider

For custom multi-provider routing with failover:

import "github.com/codewandler/llm/provider/router"

p, err := router.New(cfg, factories)

Stream Events

Events arrive as <-chan llm.Envelope where each envelope has:

type Envelope struct {
    Type EventType  // EventType constant (see below)
    Meta EventMeta  // RequestID, Seq, Timestamp
    Data any        // Polymorphic event data
}

EventType constants:

  • StreamEventCreated — emitted when stream is opened
  • StreamEventStarted — first content event with metadata (model, request_id)
  • StreamEventDelta — text, reasoning, or tool tokens
  • StreamEventToolCall — completed tool call
  • StreamEventUsageUpdated — usage update
  • StreamEventCompleted — stream finished with stop_reason
  • StreamEventError — error occurred

Event data structs:

  • DeltaEvent — text/reasoning/tool delta with Kind (Text/Reasoning/Tool)
  • ToolCallEvent — completed tool call with ToolCall tool.Call
  • StreamStartedEvent — metadata with RequestID, Model
  • CompletedEvent — StopReason
  • UsageUpdatedEvent — Usage with token counts
  • ErrorEvent — Error
Stream Processing

Use StreamProcessor with callbacks for clean event handling:

stream, err := p.CreateStream(ctx, req)
if err != nil {
    // handle error
}

result := llm.NewEventProcessor(ctx, stream).
    OnTextDelta(func(text string) { fmt.Print(text) }).
    OnReasoningDelta(func(thinking string) { /* optional */ }).
    OnStart(func(s *llm.StreamStartedEvent) { log.Printf("request %s", s.RequestID) }).
    HandleTool(tool.Handle(spec, func(ctx context.Context, p GetWeatherParams) (*GetWeatherResult, error) {
        return doWeather(p.Location, p.Unit)
    })).
    Result()

Callbacks:

  • OnTextDelta(fn func(string)) — text tokens
  • OnReasoningDelta(fn func(string)) — thinking/reasoning tokens
  • OnToolDelta(fn func(ToolDeltaPart)) — partial tool arguments
  • OnStart(fn func(*StreamStartedEvent)) — stream metadata
  • OnEvent(fn EventHandler) — all events

Result fields:

  • Text() string — accumulated text
  • Reasoning() string — accumulated thinking tokens
  • ToolCalls() []tool.Call — all tool calls
  • StopReason() StopReason — end_turn, tool_use, max_tokens, etc.
  • Usage() *Usage — token counts and cache stats
  • Message() AssistantMessage — complete response
  • Next() Messages — messages ready to append to conversation

Async tool dispatch:

llm.NewEventProcessor(ctx, ch).
    WithAsyncToolDispatch().
    HandleTool(...).
    Result()

Tool Calling

Type-Safe Tools

Define tools with tool.NewSpec[T] from github.com/codewandler/llm/tool:

import "github.com/codewandler/llm/tool"

type GetWeatherParams struct {
    Location string `json:"location" jsonschema:"description=City name,required"`
    Unit     string `json:"unit"     jsonschema:"description=Unit,enum=celsius,enum=fahrenheit"`
}

spec := tool.NewSpec[GetWeatherParams]("get_weather", "Get current weather")
Typed Dispatch
result := llm.NewEventProcessor(ctx, stream).
    HandleTool(tool.Handle(spec, func(ctx context.Context, p GetWeatherParams) (*GetWeatherResult, error) {
        return doWeather(p.Location, p.Unit)
    })).
    Result()

// Append tool result to messages
messages = append(messages, result.Next()...)
Tool Definitions (Low-Level)

For providers that need raw definitions:

tools := []llm.ToolDefinition{
    tool.DefinitionFor[GetWeatherParams]("get_weather", "Get current weather"),
}

events, err := p.CreateStream(ctx, llm.StreamRequest{
    Model:    "anthropic/claude-sonnet-4-5",
    Messages: messages,
    Tools:    tools,
})
Tool Choice
// Model decides (default)
llm.StreamRequest{ToolChoice: llm.ToolChoiceAuto{}}

// Must call at least one tool
llm.StreamRequest{ToolChoice: llm.ToolChoiceRequired{}}

// Cannot call any tools
llm.StreamRequest{ToolChoice: llm.ToolChoiceNone{}}

// Must call a specific tool
llm.StreamRequest{ToolChoice: llm.ToolChoiceTool{Name: "get_weather"}}
Type OpenAI Anthropic Ollama
ToolChoiceAuto{} "auto" {"type":"auto"} ignored
ToolChoiceRequired{} "required" {"type":"any"} ignored
ToolChoiceNone{} "none" omitted ignored
ToolChoiceTool{Name:"X"} {"type":"function",...} {"type":"tool","name":"X"} ignored
Struct Tag Reference
type Params struct {
    Location string  `json:"location" jsonschema:"description=City name,required"`
    Unit     string  `json:"unit"     jsonschema:"description=Unit,enum=celsius,enum=fahrenheit"`
    Limit    int     `json:"limit"    jsonschema:"minimum=1,maximum=100"`
    Pattern  string  `json:"pattern"  jsonschema:"pattern=^[a-z]+$"`
}

Messages

var msgs llm.Messages
msgs.AddSystemMsg("You are helpful.")
msgs.AddUserMsg("Hello")
msgs.AddAssistantMsg("Hi there")
msgs.AddToolCallResult(callID, output, false /* isError */)
msgs.Append(msg)

Or construct inline:

msgs := llm.Messages{
    &llm.SystemMsg{Content: "You are helpful."},
    &llm.UserMsg{Content: "Hello"},
    &llm.AssistantMsg{ToolCalls: []llm.ToolCall{tc}},
    &llm.ToolCallResult{ToolCallID: tc.ID, Output: result},
}

Reasoning Effort (OpenAI)

stream, _ := p.CreateStream(ctx, llm.StreamRequest{
    Model:           "openai/o3",
    Messages:        messages,
    ReasoningEffort: llm.ReasoningEffortHigh,
})

// Reasoning tokens arrive as StreamEventDelta with DeltaTypeReasoning
for event := range stream {
    if event.Type == llm.StreamEventDelta {
        fmt.Print(event.Text())          // response text
        fmt.Print(event.ReasoningText()) // thinking tokens
    }
}
Constant Value Notes
ReasoningEffortNone "none" GPT-5.1+ only
ReasoningEffortLow "low"
ReasoningEffortMedium "medium" Default for pre-5.1
ReasoningEffortHigh "high"
ReasoningEffortXHigh "xhigh" Codex-max+ only

Prompt Caching

// Top-level hint: cache the entire conversation prefix
events, err := p.CreateStream(ctx, llm.StreamRequest{
    Model:     "anthropic/claude-sonnet-4-5",
    Messages:  messages,
    CacheHint: &llm.CacheHint{Enabled: true},
})

// Per-message breakpoints (advanced)
msgs := llm.Messages{
    &llm.SystemMsg{
        Content:   largeSystemPrompt,
        CacheHint: &llm.CacheHint{Enabled: true},
    },
    &llm.UserMsg{Content: "Hello"},
}
Provider Mode TTL options
Anthropic Explicit breakpoints 5 min (default), 1 h (selected models)
Bedrock (Claude) Explicit breakpoints 5 min (default), 1 h (selected models)
OpenAI Fully automatic in-memory (default), 1 h via CacheHint{TTL:"1h"}
Ollama / OpenRouter Not supported —

Cache usage is reported in event.Usage.CacheReadTokens / CacheWriteTokens.

Model Reference Format

anthropic/claude-sonnet-4-5           # Direct Anthropic API
claude/claude-sonnet-4-5              # Claude OAuth provider
openai/gpt-4o                         # OpenAI
bedrock/anthropic.claude-3-5-sonnet   # AWS Bedrock
ollama/llama3.2:1b                    # Local Ollama
openrouter/anthropic/claude-sonnet-4.5  # OpenRouter proxy

Global aliases (configured via auto.WithGlobalAlias):

  • fast — fastest/cheapest model
  • default — balanced performance
  • powerful — most capable model
  • codex — OpenAI Codex model

Testing with llmtest

import "github.com/codewandler/llm/llmtest"

ch := llmtest.SendEvents(
    llmtest.TextEvent("hello"),
    llmtest.ToolEvent("call_1", "get_weather", map[string]any{"location": "Berlin"}),
    llmtest.DoneEvent(nil),
)

result := llm.NewEventProcessor(ctx, ch).HandleTool(...).Result()

Functions: SendEvents, TextEvent, ReasoningEvent, ToolEvent, DoneEvent, ErrorEvent.

Context Cancellation

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

events, err := p.CreateStream(ctx, llm.StreamRequest{...})
for event := range events {
    if event.Type == llm.StreamEventError {
        if errors.Is(event.Error, llm.ErrContextCancelled) {
            fmt.Println("timed out")
        }
    }
}

Environment Variables

export ANTHROPIC_API_KEY="your-api-key"
export OPENAI_KEY="your-api-key"
export OPENROUTER_API_KEY="your-api-key"
export OLLAMA_BASE_URL="http://localhost:11434"  # optional, default shown
export AWS_REGION="us-east-1"
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

Architecture

llm/
├── llm.go               # Provider interface, Streamer interface
├── event.go              # Envelope, EventType, event structs
├── event_delta.go        # DeltaEvent, TextDelta, ReasoningDelta, ToolDelta
├── event_publisher.go    # EventPublisher
├── event_handler.go      # EventHandler interface
├── event_processor.go    # StreamProcessor, NewEventProcessor, Result
├── response.go           # StopReason, Response interface
├── request.go            # StreamRequest, ReasoningEffort, OutputFormat
├── message.go            # Message types: UserMsg, AssistantMsg, ToolCallResult, etc.
├── errors.go             # ProviderError, error sentinels
├── model.go              # Model type
├── option.go             # Functional options (WithAPIKey, WithHTTPClient, etc.)
├── tool/                 # Tool types: NewSpec, Handle, TypedToolCall, Set, etc.
├── llmtest/              # Test helpers: SendEvents, TextEvent, etc.
│
└── provider/
    ├── anthropic/        # Direct Anthropic API
    │   └── claude/       # OAuth-based Claude provider
    ├── bedrock/          # AWS Bedrock
    ├── minimax/          # MiniMax API (Anthropic-compatible endpoint)
    ├── openai/           # OpenAI API (Chat + Responses API)
    ├── openrouter/       # OpenRouter proxy
    ├── ollama/           # Local Ollama
    ├── auto/             # Zero-config multi-provider setup
    ├── router/           # Multi-provider routing with failover
    └── fake/             # Test provider

CLI Tool

go run ./cmd/llmcli auth status          # Check Claude OAuth credentials
go run ./cmd/llmcli infer "Hello"        # Quick inference test
go run ./cmd/llmcli infer -v -m default "Explain Go channels"  # Verbose

Contributing

go test ./...         # run all tests
go test -race ./...   # race detector
go fmt ./...          # format
go vet ./...          # vet

See AGENTS.md for architecture and coding conventions.

License

MIT — see LICENSE.

Documentation

Index

Constants

View Source 
const (
	ProviderNameAnthropic  = "anthropic"
	ProviderNameClaude     = "claude"
	ProviderNameBedrock    = "bedrock"
	ProviderNameOllama     = "ollama"
	ProviderNameOpenAI     = "openai"
	ProviderNameOpenRouter = "openrouter"
	ProviderNameRouter     = "router"
)

Provider name constants used in ProviderError.Provider.

View Source 
const (
	RoleSystem    = msg.RoleSystem
	RoleUser      = msg.RoleUser
	RoleAssistant = msg.RoleAssistant
	RoleTool      = msg.RoleTool
	RoleDeveloper = msg.RoleDeveloper
)
View Source 
const (
	ModelDefault  = "default"
	ModelFast     = "fast"
	ModelPowerful = "powerful"
)

Variables

View Source 
var (
	// ErrContextCancelled is returned when the caller's context is cancelled
	// while a eventPub is in progress.
	ErrContextCancelled = errors.New("context cancelled")

	// ErrRequestFailed is returned when the HTTP transport fails before a
	// response is received (e.g. network error, DNS failure).
	ErrRequestFailed = errors.New("request failed")

	// ErrAPIError is returned when the provider API responds with a non-2xx
	// HTTP status. The ProviderError carries StatusCode and Body.
	ErrAPIError = errors.New("API error")

	// ErrStreamRead is returned when reading or scanning the response eventPub
	// fails at the I/O level (e.g. scanner error, connection reset).
	ErrStreamRead = errors.New("eventPub read/decode error")

	// ErrStreamDecode is returned when a eventPub chunk cannot be decoded
	// (e.g. malformed JSON in an SSE data line).
	ErrStreamDecode = errors.New("eventPub read/decode error")

	// ErrProviderError is returned when the provider sends an explicit
	// error inside the eventPub (e.g. Anthropic error event, OpenRouter
	// chunk-level error).
	ErrProviderError = errors.New("provider error")

	// ErrMissingAPIKey is returned when a provider requires an API key
	// but none has been configured.
	ErrMissingAPIKey = errors.New("missing API key")

	// ErrBuildRequest is returned when serialising the outgoing request
	// fails before it is sent.
	ErrBuildRequest = errors.New("build request error")

	// ErrUnknownModel is returned when a model ToolCallID or alias cannot be resolved.
	ErrUnknownModel = errors.New("unknown model")

	// ErrNoProviders is returned when no providers are configured or all
	// failover targets have been exhausted.
	ErrNoProviders = errors.New("no providers configured")

	// ErrUnknown is used to wrap any error that is not already a ProviderError.
	// Callers can test for it with errors.Is(err, llm.ErrUnknown).
	ErrUnknown = errors.New("unknown error")
)

Sentinel errors for use with errors.Is. Each ProviderError wraps one of these so callers can inspect the error kind without string matching.

Functions

func DefaultHttpClient added in v0.23.0 

func DefaultHttpClient() *http.Client

DefaultHttpClient returns the shared default HTTP client. It is safe for concurrent use and is reused across all providers that do not supply their own client.

func NewHttpClient added in v0.23.0 

func NewHttpClient(opts HttpClientOpts) *http.Client

NewHttpClient creates a new *http.Client with sensible defaults for LLM provider use. The client has no top-level Timeout — LLM streams can be arbitrarily long and are cancelled via context. Transport-level timeouts guard against stalled connections at the TCP/TLS layer.

When opts.Logger is non-nil, every request and response is logged at Debug level. Set opts.Debug = true to also include headers and bodies. Response bodies are tee-logged as they eventPub — no buffering, no broken SSE.

Types

type AliasResolvingProvider added in v0.29.0 

type AliasResolvingProvider struct {
	ModelResolver
	Provider
}

func ProviderWithAliasResolver added in v0.29.0 

func ProviderWithAliasResolver(p Provider) *AliasResolvingProvider

type CacheHint added in v0.20.0 

type CacheHint = msg.CacheHint

type CompletedEvent added in v0.26.0 

type CompletedEvent struct {
	StopReason StopReason `json:"stop_reason"`
}

func (CompletedEvent) Type added in v0.26.0 

func (e CompletedEvent) Type() EventType

type ContentPartEvent added in v0.29.0 

type ContentPartEvent struct {
	Part  msg.Part `json:"part"`
	Index int      `json:"index"`
}

ContentPartEvent is emitted once per content block when the provider signals block completion (content_block_stop). Index is the position of this block in the model's original output array — required to preserve the exact interleaving order of text and thinking blocks when re-serializing the assistant message.

func (ContentPartEvent) Type added in v0.29.0 

func (e ContentPartEvent) Type() EventType

type DebugEvent added in v0.26.0 

type DebugEvent struct {
	Message string `json:"message,omitempty"`
	Data    any    `json:"data,omitempty"`
}

func (DebugEvent) Type added in v0.26.0 

func (e DebugEvent) Type() EventType

type DeltaEvent added in v0.26.0 

type DeltaEvent struct {
	// Type identifies which payload field is set.
	Kind DeltaKind `json:"kind"`

	// Index is the position of this content block in the model's output array.
	// nil when the provider does not supply block-level indexing.
	//
	// Index is meaningful because a single HTTP response can contain multiple
	// blocks of the same type. Add Anthropic's interleaved-thinking beta a
	// single response may produce: thinking(0) → text(1) → tool(2) → thinking(3) → text(4).
	// Without Index a consumer cannot tell which thinking or text block a delta
	// belongs to.
	//
	// Provider semantics:
	//   Anthropic          — content_block index, all block types
	//   Bedrock            — ContentBlockIndex, all block types
	//   OpenAI Responses   — output_index, all output types
	//   OpenAI Completions — tool_calls[].index, tool calls only; text=nil
	//   OpenRouter         — tool_calls[].index, tool calls only; text=nil
	//   Ollama             — nil (complete tool calls only, no streaming fragments)
	Index *uint32 `json:"index,omitempty"`

	// Text is populated for DeltaKindText.
	Text string `json:"text,omitempty"`

	// Reasoning is populated for DeltaKindReasoning.
	Thinking string `json:"thinking,omitempty"`

	ToolDeltaPart
}

DeltaEvent carries one incremental content chunk from the model eventPub. Exactly one payload field is populated, indicated by EventType.

func TextDelta added in v0.23.0 

func TextDelta(text string) *DeltaEvent

func ThinkingDelta added in v0.29.0 

func ThinkingDelta(text string) *DeltaEvent

func ToolDelta added in v0.23.0 

func ToolDelta(id, name, argsFragment string) *DeltaEvent

func (*DeltaEvent) Type added in v0.26.0 

func (e *DeltaEvent) Type() EventType

func (*DeltaEvent) WithIndex added in v0.26.0 

func (e *DeltaEvent) WithIndex(idx uint32) *DeltaEvent

type DeltaKind added in v0.26.0 

type DeltaKind string

DeltaKind identifies the kind of incremental content carried by a DeltaEvent. 

const (
	DeltaKindText     DeltaKind = "text"
	DeltaKindThinking DeltaKind = "thinking"
	DeltaKindTool     DeltaKind = "tool"
)

type Envelope added in v0.26.0 

type Envelope struct {
	Type EventType `json:"type"`
	Meta EventMeta `json:"meta"`
	Data any       `json:"data,omitempty"`
}

type ErrorEvent added in v0.26.0 

type ErrorEvent struct {
	Error error `json:"error"`
}

func (ErrorEvent) Type added in v0.26.0 

func (e ErrorEvent) Type() EventType

type Event added in v0.26.0 

type Event interface {
	Type() EventType
}

type EventHandler added in v0.26.0 

type EventHandler interface {
	Handle(e Event)
}

type EventHandlerFunc added in v0.26.0 

type EventHandlerFunc func(e Event)

func (EventHandlerFunc) Handle added in v0.26.0 

func (h EventHandlerFunc) Handle(e Event)

type EventMeta added in v0.26.0 

type EventMeta struct {
	RequestID string            `json:"request_id,omitempty"`
	Seq       uint64            `json:"seq,omitempty"`
	CreatedAt time.Time         `json:"created_at,omitempty"`
	After     time.Duration     `json:"after,omitempty"`
	TraceID   string            `json:"trace_id,omitempty"`
	Model     string            `json:"model,omitempty"`
	Attrs     map[string]string `json:"attrs,omitempty"`
}

type EventType added in v0.26.0 

type EventType string

EventType identifies the kind of streaming event from a provider. 

const (
	StreamEventCreated      EventType = "created"
	StreamEventClosed       EventType = "closed"
	StreamEventRouted       EventType = "routed"
	StreamEventStarted      EventType = "started"
	StreamEventUsageUpdated EventType = "usage"
	StreamEventDelta        EventType = "delta"
	StreamEventToolCall     EventType = "tool_call"
	StreamEventContentPart  EventType = "content_part"
	StreamEventCompleted    EventType = "completed"
	StreamEventError        EventType = "error"
	StreamEventDebug        EventType = "debug"
)

type HttpClientOpts added in v0.23.0 

type HttpClientOpts struct {
	// Logger enables transport-level request/response logging at Debug level.
	// When nil, no logging is performed.
	Logger *slog.Logger

	// Debug extends logging to include request/response headers and bodies.
	// Has no effect when Logger is nil.
	Debug bool

	// TLSHandshakeTimeout is the maximum time allowed for a TLS handshake.
	// Defaults to 30 seconds if not set.
	TLSHandshakeTimeout time.Duration

	// ResponseHeaderTimeout is the maximum time to wait for response headers
	// after the request is sent. LLM APIs can be slow to respond (model loading,
	// queueing, cold starts), so this defaults to 120 seconds rather than the
	// typical HTTP client default.
	ResponseHeaderTimeout time.Duration
}

HttpClientOpts configures the HTTP client created by NewHttpClient.

type Message 

type Message = msg.Message

func Assistant added in v0.26.0 

func Assistant(text string) Message

func System added in v0.26.0 

func System(text string) Message

func User added in v0.26.0 

func User(text string) Message

type Messages added in v0.5.0 

type Messages = msg.Messages

type Model 

type Model struct {
	ID       string   `json:"id"`
	Name     string   `json:"name"`
	Provider string   `json:"provider"`
	Aliases  []string `json:"aliases,omitempty"`
}

Model represents an LLM model.

type ModelFetcher 

type ModelFetcher interface {
	FetchModels(ctx context.Context) ([]Model, error)
}

ModelFetcher is an optional interface providers can implement to list models dynamically from their API instead of returning a static list.

type ModelResolveFunc added in v0.29.0 

type ModelResolveFunc func(modelID string) (Model, error)

func (ModelResolveFunc) Resolve added in v0.29.0 

func (f ModelResolveFunc) Resolve(modelID string) (Model, error)

type ModelResolver added in v0.29.0 

type ModelResolver interface {
	// Resolve returns the Model for the given model ToolCallID or alias.
	// Returns an error if the model is not recognized.
	Resolve(modelID string) (Model, error)
}

ModelResolver resolves a model alias or ToolCallID to its full Model representation.

type Models added in v0.29.0 

type Models []Model

func (Models) ByAlias added in v0.29.0 

func (m Models) ByAlias(alias string) (Model, bool)

func (Models) ByID added in v0.29.0 

func (m Models) ByID(id string) (Model, bool)

func (Models) Models added in v0.29.0 

func (m Models) Models() Models

func (Models) Resolve added in v0.29.0 

func (m Models) Resolve(modelID string) (Model, error)

type ModelsProvider added in v0.29.0 

type ModelsProvider interface {
	Models() Models
}

type Named added in v0.29.0 

type Named interface {
	Name() string
}

type Option added in v0.12.0 

type Option func(*Options)

Option configures provider options.

func APIKeyFromEnv added in v0.12.0 

func APIKeyFromEnv(candidates ...string) Option

APIKeyFromEnv returns an Option that reads the API key from environment variables. It tries each candidate in order, returning the first non-empty value. Returns an error at call time if none of the candidates are set.

func WithAPIKey added in v0.12.0 

func WithAPIKey(key string) Option

WithAPIKey sets a static API key.

func WithAPIKeyFunc added in v0.12.0 

func WithAPIKeyFunc(f func(ctx context.Context) (string, error)) Option

WithAPIKeyFunc sets a dynamic API key resolver. The function is called on each CreateStream() call, enabling:

  • Lazy key resolution (fetch from secret manager on first use)
  • Key rotation (fetch fresh key each time)
  • Context-aware resolution (respect timeouts/cancellation)

func WithBaseURL added in v0.12.0 

func WithBaseURL(url string) Option

WithBaseURL sets a custom base URL for the provider.

func WithHTTPClient added in v0.23.0 

func WithHTTPClient(c *http.Client) Option

WithHTTPClient sets a custom HTTP client for the provider. When not set, providers use DefaultHttpClient().

func WithLogger added in v0.23.0 

func WithLogger(l *slog.Logger) Option

WithLogger sets a logger for providers that emit events outside the HTTP transport layer (e.g. Bedrock's binary eventstream). Events are logged at Debug level using the same format as the HTTP transport, so the same log renderer handles output from all providers.

type OptionDefaultModel added in v0.29.0 

type OptionDefaultModel struct {
	// contains filtered or unexported fields
}

func WithDefaultModel added in v0.29.0 

func WithDefaultModel() *OptionDefaultModel

type OptionModelsProvider added in v0.29.0 

type OptionModelsProvider struct {
	// contains filtered or unexported fields
}

func WithModels added in v0.29.0 

func WithModels(models Models) *OptionModelsProvider

func WithModelsProvider added in v0.29.0 

func WithModelsProvider(modelsProvider ModelsProvider) *OptionModelsProvider

type OptionMultiple added in v0.29.0 

type OptionMultiple struct {
	// contains filtered or unexported fields
}

func WithProviderOpts added in v0.29.0 

func WithProviderOpts(opts ...ProviderOpt) *OptionMultiple

type OptionStreamer added in v0.29.0 

type OptionStreamer struct {
	// contains filtered or unexported fields
}

func WithStreamer added in v0.29.0 

func WithStreamer(streamer Streamer) *OptionStreamer

type Options added in v0.12.0 

type Options struct {
	// BaseURL is the base URL for the provider's API.
	BaseURL string

	// APIKeyFunc returns the API key for authentication.
	// It is called on each CreateStream() call, allowing for lazy/dynamic resolution.
	APIKeyFunc func(ctx context.Context) (string, error)

	// HTTPClient is the HTTP client to use for API requests.
	// When nil, providers fall back to DefaultHttpClient().
	HTTPClient *http.Client

	// Logger is used by providers that cannot log via the HTTP transport
	// (e.g. Bedrock's binary eventstream). When set, eventPub events are logged
	// at Debug level using the same message format as the HTTP transport logger
	// so the same renderer handles both.
	Logger *slog.Logger
}

Options holds configuration shared across providers.

func Apply added in v0.12.0 

func Apply(opts ...Option) *Options

Apply applies all options to a new Options struct and returns it.

func (*Options) ResolveAPIKey added in v0.12.0 

func (o *Options) ResolveAPIKey(ctx context.Context) (string, error)

ResolveAPIKey calls the APIKeyFunc to get the API key. Returns an empty string (no error) if no APIKeyFunc was configured.

type OutputEffort added in v0.29.0 

type OutputEffort string

OutputEffort controls the depth/effort of the model's response (Anthropic only). Higher values result in more thorough responses at the cost of latency. 

const (
	// OutputEffortLow produces faster, less thorough responses.
	OutputEffortLow OutputEffort = "low"
	// OutputEffortMedium produces balanced responses (default).
	OutputEffortMedium OutputEffort = "medium"
	// OutputEffortHigh produces thorough, detailed responses.
	OutputEffortHigh OutputEffort = "high"
	// OutputEffortMax produces maximum effort responses (Opus 4.6 only).
	OutputEffortMax OutputEffort = "max"
)

func (OutputEffort) Valid added in v0.29.0 

func (e OutputEffort) Valid() bool

Valid returns true if the OutputEffort is a known valid value or empty.

type OutputFormat added in v0.25.0 

type OutputFormat string

OutputFormat specifies the desired output format for the model response. 

const (
	// OutputFormatText requests plain text output (default for most providers).
	OutputFormatText OutputFormat = "text"
	// OutputFormatJSON requests JSON output. The model will be constrained
	// to output valid JSON. Not all providers support this.
	OutputFormatJSON OutputFormat = "json"
)

type OverageLimit added in v0.29.0 

type OverageLimit struct {
	// Status is "allowed" or "rejected".
	Status RateLimitStatus `json:"status"`

	// DisabledReason explains why overage is disabled (e.g., "out_of_credits").
	DisabledReason string `json:"disabled_reason,omitempty"`
}

OverageLimit describes the overage (pay-as-you-go) behavior.

type Provider 

type Provider interface {
	Named
	ModelsProvider
	ModelResolver
	Streamer
}

Provider is the interface each LLM backend must implement.

func NewProvider added in v0.29.0 

func NewProvider(name string, opts ...ProviderOpt) Provider

type ProviderError added in v0.23.0 

type ProviderError struct {
	// Sentinel is one of the Err* vars above. errors.Is matches against it.
	Sentinel error `json:"-"`

	// Provider is the name of the provider that produced this error.
	// Use the ProviderName* constants.
	Provider string `json:"provider"`

	// Message is a human-readable description of the error.
	Message string `json:"message"`

	// Cause is the underlying error that triggered this one, if any.
	Cause error `json:"-"`

	// RequestBody is the raw HTTP request body. Only set for ErrBuildRequest.
	RequestBody string `json:"request_body,omitempty"`

	// StatusCode is the HTTP response status code. Only set for ErrAPIError.
	StatusCode int `json:"status_code,omitempty"`

	// Body is the raw HTTP response body. Only set for ErrAPIError.
	ResponseBody string `json:"response_body,omitempty"`
}

ProviderError is a structured error emitted by any provider. It wraps a sentinel so errors.Is works, carries the provider name for identification, and optionally holds an HTTP status code and body for API errors.

func AsProviderError added in v0.23.0 

func AsProviderError(provider string, err error) *ProviderError

AsProviderError ensures err is a *ProviderError. If it already is one, it is returned as-is. Otherwise it is wrapped in a new ProviderError with ErrUnknown as the sentinel. This guarantees that every error surface from CreateStream and EventStream.Error() is a *ProviderError.

func NewErrAPIError added in v0.23.0 

func NewErrAPIError(provider string, statusCode int, responseBody string) *ProviderError

NewErrAPIError wraps a non-2xx HTTP response from a provider API.

func NewErrAPIErrorWithRequest added in v0.29.0 

func NewErrAPIErrorWithRequest(provider string, requestBody string, statusCode int, responseBody string) *ProviderError

NewErrAPIErrorWithRequest wraps a non-2xx HTTP response from a provider API.

func NewErrBuildRequest added in v0.23.0 

func NewErrBuildRequest(provider string, cause error) *ProviderError

NewErrBuildRequest wraps a failure that occurred while building the outgoing request (e.g. JSON serialisation error).

func NewErrContextCancelled added in v0.23.0 

func NewErrContextCancelled(provider string, cause error) *ProviderError

NewErrContextCancelled wraps a context cancellation for a provider eventPub.

func NewErrMissingAPIKey added in v0.23.0 

func NewErrMissingAPIKey(provider string) *ProviderError

NewErrMissingAPIKey returns an error for a provider that has no API key configured.

func NewErrNoProviders added in v0.23.0 

func NewErrNoProviders(provider string) *ProviderError

NewErrNoProviders returns an error when no providers are available or all failover targets have been exhausted.

func NewErrProviderMsg added in v0.23.0 

func NewErrProviderMsg(provider string, msg string) *ProviderError

NewErrProviderMsg wraps an explicit error message sent by the provider inside the eventPub (e.g. an Anthropic error event or OpenRouter chunk error).

func NewErrRequestFailed added in v0.23.0 

func NewErrRequestFailed(provider string, cause error) *ProviderError

NewErrRequestFailed wraps an HTTP transport-level failure.

func NewErrStreamDecode added in v0.23.0 

func NewErrStreamDecode(provider string, cause error) *ProviderError

NewErrStreamDecode wraps a JSON or protocol decode failure mid-eventPub.

func NewErrStreamRead added in v0.23.0 

func NewErrStreamRead(provider string, cause error) *ProviderError

NewErrStreamRead wraps an I/O or scanner error that occurred while reading the response eventPub.

func NewErrUnknownModel added in v0.23.0 

func NewErrUnknownModel(provider string, modelID string) *ProviderError

NewErrUnknownModel returns an error for a model ToolCallID or alias that cannot be resolved by the provider.

func (*ProviderError) Error added in v0.23.0 

func (e *ProviderError) Error() string

Error returns a human-readable error string in the form: "<provider>: <sentinel>: <message>" or "<provider>: <sentinel>: <message>: <cause>".

func (*ProviderError) Is added in v0.23.0 

func (e *ProviderError) Is(target error) bool

Is reports whether this error matches target. It matches if target is the same sentinel, enabling errors.Is(err, ErrAPIError) etc.

func (*ProviderError) MarshalJSON added in v0.23.0 

func (e *ProviderError) MarshalJSON() ([]byte, error)

MarshalJSON serialises ProviderError to JSON. Sentinel and Cause are rendered as strings so the full error is machine-readable.

func (*ProviderError) Unwrap added in v0.23.0 

func (e *ProviderError) Unwrap() error

Unwrap returns Cause when set, allowing errors.As/Is to traverse the chain. When Cause is nil, Unwrap returns Sentinel so errors.Is(err, ErrAPIError) still works even with no underlying cause.

func (*ProviderError) WithRequestBody added in v0.29.0 

func (e *ProviderError) WithRequestBody(body string) *ProviderError

type ProviderOpt added in v0.29.0 

type ProviderOpt interface {
	// contains filtered or unexported methods
}

type Publisher added in v0.26.0 

type Publisher interface {
	Publish(payload Event)

	Started(started StreamStartedEvent)
	Routed(routed RouteInfo)
	Delta(d *DeltaEvent)
	ToolCall(tc tool.Call)
	ContentBlock(evt ContentPartEvent)

	Usage(usage Usage)
	Completed(completed CompletedEvent)

	Error(err error)
	Debug(msg string, data any)

	Close()
}

func NewEventPublisher added in v0.26.0 

func NewEventPublisher() (Publisher, <-chan Envelope)

type RateLimitStatus added in v0.29.0 

type RateLimitStatus string

RateLimitStatus represents the status of a rate limit. 

const (
	RateLimitStatusAllowed    RateLimitStatus = "allowed"
	RateLimitStatusOverBudget RateLimitStatus = "over_budget"
	RateLimitStatusBlocked    RateLimitStatus = "blocked"
)

type RateLimits added in v0.29.0 

type RateLimits struct {
	// Unified limits (applies to the unified API endpoint).
	Unified *UnifiedRateLimit `json:"unified,omitempty"`

	// OrganizationID is the org this request was made under.
	OrganizationID string `json:"organization_id,omitempty"`

	// RequestID is the upstream request identifier.
	RequestID string `json:"request_id,omitempty"`
}

RateLimits holds parsed rate-limit headers from Anthropic API responses. These are emitted in the StreamStartedEvent so consumers can inspect them.

func ParseRateLimits added in v0.29.0 

func ParseRateLimits(headers map[string]string) *RateLimits

ParseRateLimits parses rate-limit headers from an Anthropic HTTP response. Pass the response headers map (lowercased keys → values).

type Request added in v0.25.0 

type Request struct {
	// Model is the model identifier or alias to use, e.g. "fast", "anthropic/claude-sonnet-4-5".
	Model string `json:"model"`

	// Messages is the conversation history to send to the model.
	Messages Messages `json:"messages"`

	// MaxTokens limits the maximum number of tokens in the response.
	// When 0, the provider's default is used.
	MaxTokens int `json:"max_tokens,omitempty"`

	// Temperature controls randomness in sampling. Higher values produce
	// more diverse outputs (0.0-2.0 for most providers). Not supported by
	// Anthropic.
	Temperature float64 `json:"temperature,omitempty"`

	// TopP is the nucleus sampling threshold. The model considers only tokens
	// comprising the top P probability mass. Not supported by Anthropic.
	TopP float64 `json:"top_p,omitempty"`

	// TopK restricts token selection to the K most likely tokens. Higher values
	// increase diversity. Not supported by Anthropic.
	TopK int `json:"top_k,omitempty"`

	// OutputFormat specifies the desired output format.
	// Supported by OpenAI and Anthropic. When set to JSON, the model will
	// be constrained to output valid JSON.
	OutputFormat OutputFormat `json:"output_format,omitempty"`

	// Tools is the set of tools the model may call during the response.
	Tools []llmtool.Definition `json:"tools,omitempty"`

	// ToolChoice controls how the model selects tools. Defaults to Auto when Tools are provided.
	ToolChoice ToolChoice `json:"tool_choice,omitempty"`

	// ThinkingEffort controls the depth of reasoning for models that support it (e.g. OpenAI o-series).
	ThinkingEffort ThinkingEffort `json:"thinking_effort,omitempty"`

	// OutputEffort controls the depth/effort of the model's response (Anthropic only).
	OutputEffort OutputEffort `json:"output_effort,omitempty"`

	// CacheHint is a top-level prompt caching hint. Behaviour is provider-specific:
	// Anthropic auto mode, Bedrock trailing cachePoint, OpenAI extended retention.
	CacheHint *CacheHint `json:"cache_hint,omitempty"`
}

Request configures a provider CreateStream call.

func (Request) Validate added in v0.25.0 

func (o Request) Validate() error

Validate checks that the options are valid.

type Response added in v0.26.0 

type Response interface {
	Message() msg.Message
	Text() string
	Thought() string
	StopReason() StopReason
	Usage() *Usage
	Error() error
	ToolCalls() []tool.Call
	ToolResults() []tool.Result
}

type Result added in v0.26.0 

type Result interface {
	Response
	Next() msg.Messages
}

func ProcessEvents added in v0.26.0 

func ProcessEvents(ctx context.Context, ch <-chan Envelope) Result

type Role 

type Role = msg.Role

type RouteInfo added in v0.26.0 

type RouteInfo struct {
	Provider       string  `json:"provider"`
	ModelRequested string  `json:"model_requested,omitempty"`
	ModelResolved  string  `json:"model_resolved,omitempty"`
	Errors         []error `json:"-"`
}

type RouteInfoEvent added in v0.26.0 

type RouteInfoEvent struct {
	RouteInfo RouteInfo `json:"route_info"`
}

func (RouteInfoEvent) Type added in v0.26.0 

func (e RouteInfoEvent) Type() EventType

type StopReason added in v0.23.0 

type StopReason string

StopReason describes why the model stopped generating. 

const (
	// StopReasonEndTurn is natural completion — the model finished its response.
	StopReasonEndTurn StopReason = "end_turn"
	// StopReasonToolUse means the model emitted one or more tool calls.
	StopReasonToolUse StopReason = "tool_use"
	// StopReasonMaxTokens means the output length limit was reached.
	StopReasonMaxTokens StopReason = "max_tokens"
	// StopReasonContentFilter means output was blocked by the provider.
	StopReasonContentFilter StopReason = "content_filter"
	// StopReasonCancelled means the context was cancelled before the eventPub ended.
	StopReasonCancelled StopReason = "cancelled"
	// StopReasonError means the eventPub ended with a StreamEventError.
	StopReasonError StopReason = "error"

	StopReasonUnknown StopReason = ""
)

type Stream added in v0.26.0 

type Stream <-chan Envelope

type StreamClosedEvent added in v0.26.0 

type StreamClosedEvent struct{}

func (StreamClosedEvent) Type added in v0.26.0 

func (e StreamClosedEvent) Type() EventType

type StreamCreatedEvent added in v0.26.0 

type StreamCreatedEvent struct{}

func (StreamCreatedEvent) Type added in v0.26.0 

func (e StreamCreatedEvent) Type() EventType

type StreamFunc added in v0.29.0 

type StreamFunc func(ctx context.Context, opts Request) (Stream, error)

func (StreamFunc) CreateStream added in v0.29.0 

func (f StreamFunc) CreateStream(ctx context.Context, opts Request) (Stream, error)

type StreamProcessor added in v0.26.0 

type StreamProcessor struct {
	// contains filtered or unexported fields
}

func NewEventProcessor added in v0.26.0 

func NewEventProcessor(ctx context.Context, ch <-chan Envelope) *StreamProcessor

func (*StreamProcessor) HandleTool added in v0.26.0 

func (r *StreamProcessor) HandleTool(handlers ...tool.NamedHandler) *StreamProcessor

HandleTool registers a Handler that is invoked when the model emits a completed tool call matching h.ToolName(). The handler's output is stored in StreamResult.ToolResults and included in the messages returned by Next/Apply.

Pass a *BoundToolSpec (from llm.Handle) for typed, spec-aware handlers: 

proc.HandleTool(llm.Handle(weatherSpec, func(ctx context.Context, in GetWeatherParams) (*GetWeatherResult, error) {
    return &GetWeatherResult{Temp: 22}, nil
}))

func (*StreamProcessor) OnDelta added in v0.26.0 

func (r *StreamProcessor) OnDelta(fn TypedEventHandler[*DeltaEvent]) *StreamProcessor

func (*StreamProcessor) OnEvent added in v0.26.0 

func (r *StreamProcessor) OnEvent(fn EventHandler) *StreamProcessor

func (*StreamProcessor) OnReasoningDelta added in v0.26.0 

func (r *StreamProcessor) OnReasoningDelta(fn func(delta string)) *StreamProcessor

OnReasoningDelta registers a callback that is called for each incremental reasoning/thinking token.

func (*StreamProcessor) OnStart added in v0.26.0 

func (r *StreamProcessor) OnStart(fn TypedEventHandler[*StreamStartedEvent]) *StreamProcessor

OnStart registers a callback that is called when the StreamEventStarted event arrives, carrying provider metadata (request ToolCallID, model, time-to-first-token).

func (*StreamProcessor) OnTextDelta added in v0.26.0 

func (r *StreamProcessor) OnTextDelta(fn func(delta string)) *StreamProcessor

OnTextDelta registers a callback that is called for each incremental text token. Panics in the callback are recovered and recorded on the StreamResult error.

func (*StreamProcessor) OnToolDelta added in v0.26.0 

func (r *StreamProcessor) OnToolDelta(fn func(d ToolDeltaPart)) *StreamProcessor

OnToolDelta registers a callback that is called for each partial tool-call argument fragment (DeltaKindTool deltas).

func (*StreamProcessor) Result added in v0.26.0 

func (r *StreamProcessor) Result() Result

Result starts consuming the eventPub (at most once) and returns a channel that yields exactly one *StreamResult when the eventPub is fully processed. The channel is closed after the result is sent.

Calling Result() multiple times is safe — the eventPub is only consumed once and the same channel is returned on subsequent calls.

func (*StreamProcessor) WithAsyncToolDispatch added in v0.26.0 

func (r *StreamProcessor) WithAsyncToolDispatch() *StreamProcessor

WithAsyncToolDispatch switches tool handler dispatch to concurrent mode: all tool calls emitted in a single response are executed in parallel, one goroutine per call. Results are collected in emission order before the eventPub is considered complete.

func (*StreamProcessor) WithToolDispatcher added in v0.26.0 

func (r *StreamProcessor) WithToolDispatcher(d tool.DispatcherType) *StreamProcessor

WithToolDispatcher sets the tool dispatcher explicitly.

type StreamRequest added in v0.23.0 

type StreamRequest = Request

type StreamStartedEvent added in v0.26.0 

type StreamStartedEvent struct {
	RequestID string `json:"request_id,omitempty"`

	// Model is the model identifier returned by the upstream API in its response.
	// e.g., "claude-haiku-4-5-20251001". May be empty if the API doesn't echo the model back.
	Model string `json:"model,omitempty"`

	// Extra holds provider-specific data such as rate-limit headers.
	Extra map[string]any `json:"extra,omitempty"`
}

func (StreamStartedEvent) Type added in v0.26.0 

func (e StreamStartedEvent) Type() EventType

type Streamer added in v0.19.0 

type Streamer interface {
	CreateStream(ctx context.Context, opts Request) (Stream, error)
}

type ThinkingEffort added in v0.29.0 

type ThinkingEffort string

ThinkingEffort controls the amount of reasoning for reasoning models. Lower values result in faster responses with fewer reasoning tokens. 

const (
	// ThinkingEffortNone disables reasoning (GPT-5.1+ only).
	ThinkingEffortNone ThinkingEffort = "none"
	// ThinkingEffortMinimal uses minimal reasoning effort.
	ThinkingEffortMinimal ThinkingEffort = "minimal"
	// ThinkingEffortLow uses low reasoning effort.
	ThinkingEffortLow ThinkingEffort = "low"
	// ThinkingEffortMedium uses medium reasoning effort (default for most models before GPT-5.1).
	ThinkingEffortMedium ThinkingEffort = "medium"
	// ThinkingEffortHigh uses high reasoning effort.
	ThinkingEffortHigh ThinkingEffort = "high"
	// ThinkingEffortXHigh uses extra high reasoning effort (codex-max+ only).
	ThinkingEffortXHigh ThinkingEffort = "xhigh"
)

func (ThinkingEffort) Valid added in v0.29.0 

func (r ThinkingEffort) Valid() bool

Valid returns true if the ThinkingEffort is a known valid value or empty.

type ToolCallEvent added in v0.26.0 

type ToolCallEvent struct {
	ToolCall tool.Call `json:"tool_call"`
}

func (ToolCallEvent) Type added in v0.26.0 

func (e ToolCallEvent) Type() EventType

type ToolChoice added in v0.6.0 

type ToolChoice interface {
	String() string
	// contains filtered or unexported methods
}

type ToolChoiceAuto added in v0.6.0 

type ToolChoiceAuto struct{}

func (ToolChoiceAuto) String added in v0.29.0 

func (t ToolChoiceAuto) String() string

type ToolChoiceNone added in v0.6.0 

type ToolChoiceNone struct{}

func (ToolChoiceNone) String added in v0.29.0 

func (t ToolChoiceNone) String() string

type ToolChoiceRequired added in v0.6.0 

type ToolChoiceRequired struct{}

func (ToolChoiceRequired) String added in v0.29.0 

func (t ToolChoiceRequired) String() string

type ToolChoiceTool added in v0.6.0 

type ToolChoiceTool struct {
	Name string
}

func (ToolChoiceTool) String added in v0.29.0 

func (t ToolChoiceTool) String() string

type ToolDeltaPart added in v0.26.0 

type ToolDeltaPart struct {
	// ToolID, ToolName, and ToolArgs are populated for DeltaKindTool.
	// ToolArgs is a raw partial JSON fragment — not yet a complete object.
	ToolID   string `json:"tool_id,omitempty"`
	ToolName string `json:"tool_name,omitempty"`
	ToolArgs string `json:"tool_args,omitempty"`
}

type TypedEventHandler added in v0.26.0 

type TypedEventHandler[T any] func(e T)

func (TypedEventHandler[T]) Handle added in v0.26.0 

func (h TypedEventHandler[T]) Handle(e Event)

type UnifiedRateLimit added in v0.29.0 

type UnifiedRateLimit struct {
	// The current status: "allowed", "over_budget", or "blocked".
	Status RateLimitStatus `json:"status"`

	// ResetAt is the Unix timestamp when the primary window resets.
	ResetAt time.Time `json:"reset_at"`

	// FiveHour contains limits for the 5-hour rolling window.
	FiveHour *WindowLimit `json:"five_hour,omitempty"`

	// SevenDay contains limits for the 7-day rolling window.
	SevenDay *WindowLimit `json:"seven_day,omitempty"`

	// Overage describes whether overage usage is enabled and why it might be disabled.
	Overage *OverageLimit `json:"overage,omitempty"`

	// FallbackPercentage is between 0 and 1, indicating how much of the fallback
	// (pay-as-you-go) pool is being used when the primary budget is exhausted.
	FallbackPercentage float64 `json:"fallback_percentage,omitempty"`

	// RepresentativeClaim identifies which tier/bucket this request counts against.
	RepresentativeClaim string `json:"representative_claim,omitempty"`
}

UnifiedRateLimit contains the unified rate-limit data from the Anthropic-Ratelimit-Unified-* headers.

type Usage 

type Usage struct {
	// InputTokens is the total number of input tokens processed, including
	// tokens served from cache (CacheReadTokens) and tokens written to cache
	// (CacheWriteTokens). Callers can use this as the single "how many input
	// tokens did this request consume" figure.
	InputTokens int `json:"input_tokens"`

	// OutputTokens is the number of tokens generated in the response.
	OutputTokens int `json:"output_tokens"`

	// TotalTokens is InputTokens + OutputTokens.
	TotalTokens int `json:"total_tokens"`

	// Cost is the total request cost in USD.
	// For Anthropic, Bedrock, and OpenAI this is locally calculated from
	// provider pricing tables and equals the sum of the breakdown fields below.
	// For OpenRouter this is API-reported by the proxy (already includes cache pricing).
	Cost float64 `json:"cost"`

	// Detailed token breakdown (provider-specific, may be zero).
	CacheReadTokens  int `json:"cache_read_tokens,omitempty"`  // Input tokens served from an existing cache entry (all providers).
	CacheWriteTokens int `json:"cache_write_tokens,omitempty"` // Input tokens written to a new cache entry (Anthropic, Bedrock).
	ReasoningTokens  int `json:"reasoning_tokens,omitempty"`   // ToolOutput tokens consumed by model reasoning (e.g. extended thinking).

	// Granular cost breakdown in USD (zero if provider/model pricing is unknown).
	// Sum of InputCost + CacheReadCost + CacheWriteCost + OutputCost == Cost.
	// Not populated for OpenRouter (API-reported cost is used instead).
	//
	// InputCost covers only the non-cached, non-write portion:
	// InputTokens - CacheReadTokens - CacheWriteTokens tokens at the regular input rate.
	InputCost      float64 `json:"input_cost,omitempty"`       // Cost of non-cached, non-write input tokens.
	CacheReadCost  float64 `json:"cache_read_cost,omitempty"`  // Cost of cache-read tokens.
	CacheWriteCost float64 `json:"cache_write_cost,omitempty"` // Cost of cache-write tokens.
	OutputCost     float64 `json:"output_cost,omitempty"`      // Cost of output tokens.
}

Usage holds token counts and cost from a provider response.

type UsageUpdatedEvent added in v0.26.0 

type UsageUpdatedEvent struct {
	Usage Usage `json:"usage"`
}

func (UsageUpdatedEvent) Type added in v0.26.0 

func (e UsageUpdatedEvent) Type() EventType

type WindowLimit added in v0.29.0 

type WindowLimit struct {
	// Status is "allowed" or "blocked".
	Status RateLimitStatus `json:"status"`

	// ResetAt is the Unix timestamp when this window resets.
	ResetAt time.Time `json:"reset_at"`

	// Utilization is between 0 and 1, representing how much of this window is used.
	Utilization float64 `json:"utilization"`
}

WindowLimit represents a rate-limit window (e.g., 5-hour or 7-day).

Source Files

View all Source files

Directories

Path Synopsis
cmd
llmcli command
llmcli is a command-line tool for testing LLM providers.
 llmcli is a command-line tool for testing LLM providers.
llmcli/cmds
Package cmds provides CLI commands for llmcli.
 Package cmds provides CLI commands for llmcli.
llmcli/store
Package store provides token storage implementations.
 Package store provides token storage implementations.
internal
sse
Package llmtest provides helpers for testing code that consumes llm.Stream channels, following the convention of packages like net/http/httptest.
 Package llmtest provides helpers for testing code that consumes llm.Stream channels, following the convention of packages like net/http/httptest.
Package modeldb provides access to the models.dev model database.
 Package modeldb provides access to the models.dev model database.
provider
anthropic
anthropic/claude
Package claude provides an Anthropic provider using Claude OAuth tokens.
 Package claude provides an Anthropic provider using Claude OAuth tokens.
auto
Package auto provides zero-config multi-provider setup for LLM providers.
 Package auto provides zero-config multi-provider setup for LLM providers.
bedrock
fake
minimax
Package minimax provides a MiniMax LLM provider using the Anthropic-compatible API.
 Package minimax provides a MiniMax LLM provider using the Anthropic-compatible API.
ollama
openai
openrouter
router
Package tokencount provides a shared offline tiktoken wrapper for LLM token estimation.
 Package tokencount provides a shared offline tiktoken wrapper for LLM token estimation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.