llm

package
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package llm is a unified, provider-neutral API for large language models.

It speaks two wire protocols, OpenAI Chat Completions and Anthropic Messages, behind one set of types. The same conversation can be sent to any model on either protocol, and the target model can change between turns; the library re-adapts the history for each request. It is a stateless translation layer: it decides what to send for a request and how to interpret the streamed response, and leaves history storage, context compaction, and tool-loop orchestration to the caller.

Entry points

Stream returns a channel of Event values; Complete consumes that stream and returns the final AssistantMessage. Both dispatch through the package default client to the adapter registered for the model's protocol.

Protocol adapters live in provider sub-packages and register themselves on import. Pull in the protocols an application needs — and only their vendor SDKs — by importing the matching provider package for its side effects, or import github.com/ktsoator/or/llm/all for every built-in protocol at once: 

import (
	"github.com/ktsoator/or/llm"
	_ "github.com/ktsoator/or/llm/anthropic"
)

model := llm.GetModel("anthropic", "claude-opus-4-8")
msg, err := llm.Complete(ctx, model, llm.Prompt("hello"), llm.StreamOptions{})

A caller that prefers explicit wiring can build its own registry and client with NewRegistry, Registry.Register, and NewClient instead of the default.

Building a request

A Context holds the system prompt, the message history, and the available tools. Messages are provider-neutral and serialize to self-describing JSON, so a conversation can be persisted and replayed later against any model:

Text-only models that receive image content have it downgraded to a placeholder automatically. The Prompt, UserText, and related helpers build the common text-only cases without the nesting boilerplate.

Options

StreamOptions carries settings shared by every protocol — API key, temperature, max tokens, headers, retries, timeout — plus observation hooks, OnRequest (the exact serialized request body) and OnResponse (status and headers), and RewriteRequest, which replaces the serialized body before it is sent. Each is invoked once per attempt including retries.

Reasoning is a provider-neutral effort level (ModelThinkingLevel: off, minimal, low, medium, high, xhigh). Each adapter maps it to that provider's native form (Anthropic adaptive or budget thinking; the OpenAI-compatible reasoning fields) and clamps it to what the model supports. It is ignored by non-reasoning models.

Settings with no neutral equivalent live on a protocol-specific extension supplied through StreamOptions.ProtocolOptions and validated against the target protocol before the request is sent:

Streaming

Event values report incremental progress: text, reasoning, and tool-call blocks each emit start, delta, and end events, followed by a single terminal EventDone (carrying the final message) or EventError. Every event carries a Partial snapshot of the message assembled so far. Tool-call arguments are parsed best-effort at end of stream; validate them and wait for EventDone before executing any call.

Typed tools

NewTool and MustTool derive a provider-compatible JSON Schema from a Go struct, and DecodeToolCall decodes a returned call back into that struct. Malformed or truncated argument JSON degrades to a best-effort value and is reported in AssistantMessage.Diagnostics rather than failing the response.

Results

AssistantMessage holds the response content, a StopReason, token Usage with per-category cost, and any non-fatal Diagnostics. CalculateCost prices a usage record against a model.

Switching models

TransformMessages adapts a stored history for a target model before replay: it downgrades unsupported images, preserves reasoning signatures for the same model while downgrading or dropping them across models, normalizes tool-call identifiers, and repairs unanswered tool calls. Stream and Complete apply it automatically. IsContextOverflow reports whether a response exceeded the model's context window.

Models

LookupModel and GetModel resolve models from the built-in catalog; GetProviders and GetModels enumerate it. SupportedThinkingLevels and ClampThinkingLevel report and adjust a model's reasoning levels. A caller may also construct a Model directly, pointing BaseURL at any OpenAI-compatible or Anthropic-compatible endpoint.

Custom protocols

A genuinely different wire protocol is added by implementing ProtocolAdapter and registering it — with Register for the package default registry, or with Registry.Register on a registry passed to NewClient. NewStreamWriter gives the adapter the same event-stream machinery the built-ins use.

Index

Constants

View Source 
const DiagnosticToolArgumentsRecovered = "tool_arguments_recovered"

DiagnosticToolArgumentsRecovered is the Type of a diagnostic recorded when a tool call's arguments could not be parsed strictly and were repaired, partially recovered, or discarded.

Variables

This section is empty.

Functions

func APIKeyEnvVars added in v0.5.2 

func APIKeyEnvVars(provider string) []string

APIKeyEnvVars returns the environment variables checked for provider in precedence order. The returned slice is safe for the caller to modify.

func DecodeToolCall 

func DecodeToolCall[T any](tool ToolDefinition, call ToolCall) (T, error)

DecodeToolCall validates and coerces call arguments with tool's schema, then decodes them into T. The original ToolCall is not modified.

func FindEnvAPIKeys added in v0.5.2 

func FindEnvAPIKeys(provider string) []string

FindEnvAPIKeys returns the names of configured API key environment variables for provider, in lookup order.

func FindEnvAPIKeysWithEnv added in v0.5.2 

func FindEnvAPIKeysWithEnv(provider string, env ProviderEnv) []string

FindEnvAPIKeysWithEnv is FindEnvAPIKeys with request-scoped overrides.

func GetEnvAPIKey added in v0.5.2 

func GetEnvAPIKey(provider string) string

GetEnvAPIKey returns the first configured API key for provider.

func GetEnvAPIKeyWithEnv added in v0.5.2 

func GetEnvAPIKeyWithEnv(provider string, env ProviderEnv) string

GetEnvAPIKeyWithEnv returns the first configured API key for provider, preferring non-empty request-scoped values over process environment values.

func GetProviders 

func GetProviders() []string

GetProviders returns all providers in the built-in model registry.

func IsContextOverflow 

func IsContextOverflow(message AssistantMessage, contextWindow int64) bool

IsContextOverflow reports whether an assistant message indicates that the input exceeded the model's context window. It covers three cases:

  1. Error-based overflow: most providers return StopReasonError with a recognizable error message (matched against overflowPatterns, excluding nonOverflowPatterns such as rate limits).
  2. Silent overflow (e.g. z.ai): the request succeeds but usage.Input exceeds the context window. Pass a non-zero contextWindow to detect this.
  3. Length-stop overflow (e.g. Xiaomi MiMo): the server truncates oversized input to fill the window, leaving no room to generate, so it returns StopReasonLength with zero output and input filling the window.

Pass contextWindow as the model's window size to enable cases 2 and 3; pass 0 to check error messages only.

func OverflowPatterns added in v0.5.2 

func OverflowPatterns() []*regexp.Regexp

OverflowPatterns returns a copy of the overflow detection patterns, primarily for tests.

func ParseToolArguments 

func ParseToolArguments(raw string) map[string]any

ParseToolArguments decodes a tool call's accumulated JSON arguments into an object on a best-effort basis. Models occasionally emit JSON with unescaped control characters or bad escapes, and a stream may be cut off mid-token, so a strict decode of that input would lose every argument. This therefore tries, in order: a strict decode, a decode of a repaired copy, a partial decode that closes any open containers and strings, and a partial decode of the repaired copy. It always returns a non-nil map, falling back to an empty object when nothing can be salvaged.

Parsing never fails the surrounding stream: a recoverable but invalid tool call surfaces with whatever arguments could be salvaged so an agent can still validate it (see ValidateToolArguments) and let the model self-correct, instead of aborting the whole response.

func Register added in v0.5.2 

func Register(adapter ProtocolAdapter) error

Register adds adapter to the package default registry, replacing any adapter already registered for its protocol. Provider packages call it from init; most applications register a provider by importing its package rather than calling Register directly.

func Stream 

func Stream(ctx context.Context, model Model, input Context, options StreamOptions) (<-chan Event, error)

Stream starts a streaming model request using the default client. The model's protocol must have an adapter registered (import the matching provider package); otherwise it returns a "no adapter registered" error.

func ValidateToolArguments 

func ValidateToolArguments(tool ToolDefinition, toolCall ToolCall) (map[string]any, error)

ValidateToolArguments coerces a tool call's arguments toward the tool's JSON Schema (forgiving common model mistakes such as "3" for a number), then validates them. It returns the coerced arguments, or a detailed error naming the failing fields. The original toolCall.Arguments are left unchanged.

The coercion is a best-effort pass toward the schema. Validation covers the JSON Schema features normally emitted for tool definitions, including composition keywords and object, array, string, and numeric constraints.

func ValidateToolCall 

func ValidateToolCall(tools []ToolDefinition, toolCall ToolCall) (map[string]any, error)

ValidateToolCall finds the tool named by the call and validates its arguments. It is a utility callers may invoke before dispatching a tool, not something the library calls itself. It returns the coerced arguments.

Types

type AnthropicMessagesCompatibility 

type AnthropicMessagesCompatibility struct {
	SupportsTemperature       *bool `json:"supportsTemperature,omitempty"`
	SupportsCacheControl      *bool `json:"supportsCacheControl,omitempty"`
	SupportsCacheControlTools *bool `json:"supportsCacheControlOnTools,omitempty"`
	ForceAdaptiveThinking     *bool `json:"forceAdaptiveThinking,omitempty"`
	AllowEmptySignature       *bool `json:"allowEmptySignature,omitempty"`
}

AnthropicMessagesCompatibility describes differences between providers that implement an Anthropic Messages-compatible endpoint. Pointer booleans distinguish an explicit false value from an unspecified provider default. Anthropic-compatible vendors (e.g. MiniMax) are served by pointing the base URL at their endpoint; most need no overrides at all.

func (*AnthropicMessagesCompatibility) Protocol added in v0.5.2 

func (*AnthropicMessagesCompatibility) Protocol() Protocol

Protocol identifies the API protocol whose request and message dialect this compatibility configuration describes.

type AnthropicStreamOptions added in v0.2.0 

type AnthropicStreamOptions struct {
	// ThinkingDisplay controls how a reasoning model returns its thinking. Empty
	// defaults to summarized.
	ThinkingDisplay ThinkingDisplay
	// ToolChoice controls Anthropic's native tool selection behavior.
	ToolChoice AnthropicToolChoice
}

AnthropicStreamOptions contains settings understood only by the Anthropic Messages protocol. Keeping them nested prevents provider-specific knobs from flattening the shared StreamOptions namespace.

func (*AnthropicStreamOptions) Protocol added in v0.5.2 

func (*AnthropicStreamOptions) Protocol() Protocol

Protocol identifies the protocol that accepts these options.

func (*AnthropicStreamOptions) Validate added in v0.5.2 

func (options *AnthropicStreamOptions) Validate(tools []ToolDefinition) error

Validate checks the Anthropic-specific settings against the available tools.

type AnthropicToolChoice added in v0.2.0 

type AnthropicToolChoice interface {
	// contains filtered or unexported methods
}

AnthropicToolChoice is the native Anthropic tool_choice union. Use one of the AnthropicToolChoice* mode constants or AnthropicToolChoiceTool.

type AnthropicToolChoiceMode added in v0.2.0 

type AnthropicToolChoiceMode string

AnthropicToolChoiceMode is one of Anthropic's string tool-choice modes. 

const (
	AnthropicToolChoiceAuto AnthropicToolChoiceMode = "auto"
	AnthropicToolChoiceAny  AnthropicToolChoiceMode = "any"
	AnthropicToolChoiceNone AnthropicToolChoiceMode = "none"
)

type AnthropicToolChoiceTool added in v0.2.0 

type AnthropicToolChoiceTool struct {
	Name string
}

AnthropicToolChoiceTool forces the model to call Name.

type ArgumentsMode added in v0.2.0 

type ArgumentsMode string

ArgumentsMode reports which layer of ParseToolArgumentsMode produced a result, so callers can tell strictly parsed arguments apart from recovered ones. 

const (
	// ArgumentsStrict means empty input or a clean strict decode: fully trusted.
	ArgumentsStrict ArgumentsMode = "strict"
	// ArgumentsRepaired means the input decoded only after escape repair.
	ArgumentsRepaired ArgumentsMode = "repaired"
	// ArgumentsPartial means truncated input was closed and decoded, so some
	// fields may be missing or cut short.
	ArgumentsPartial ArgumentsMode = "partial"
	// ArgumentsInvalid means nothing could be salvaged and the result is empty.
	ArgumentsInvalid ArgumentsMode = "invalid"
)

func ParseToolArgumentsMode added in v0.2.0 

func ParseToolArgumentsMode(raw string) (map[string]any, ArgumentsMode)

ParseToolArgumentsMode is ParseToolArguments with the recovery mode it used. The arguments are identical to ParseToolArguments; the mode lets a caller decline to execute a tool whose arguments were not strictly parsed.

type AssistantContent 

type AssistantContent interface {
	// contains filtered or unexported methods
}

AssistantContent is content that can appear in an assistant message.

type AssistantMessage 

type AssistantMessage struct {
	// Content contains the model output blocks: text, thinking, and tool calls.
	Content []AssistantContent `json:"content"`

	// --- Response metadata ---
	// Protocol is the wire protocol used for this response.
	Protocol Protocol `json:"protocol"`
	// Provider is the model provider that produced this response.
	Provider string `json:"provider"`
	// Model is the requested model ID.
	Model string `json:"model"`
	// ResponseModel is the model name reported by the provider, when it differs
	// from or further qualifies the requested model.
	ResponseModel string `json:"responseModel,omitempty"`
	// ResponseID is the provider's unique identifier for this response.
	ResponseID string `json:"responseId,omitempty"`
	// Usage records token consumption and calculated cost for this response.
	Usage Usage `json:"usage"`
	// StopReason explains why generation stopped.
	StopReason StopReason `json:"stopReason"`
	// ErrorMessage stores the provider or runtime error for failed responses.
	ErrorMessage string `json:"errorMessage,omitempty"`
	// Diagnostics records non-fatal events (failures recovered from, degraded
	// results) that occurred while producing this response. It is nil for a
	// clean response.
	Diagnostics []Diagnostic `json:"diagnostics,omitempty"`
	// Timestamp is the Unix millisecond time when the response object was
	// created.
	Timestamp int64 `json:"timestamp"`
}

AssistantMessage is the final or partial response returned by a provider.

func AssistantText added in v0.5.1 

func AssistantText(text string) *AssistantMessage

AssistantText builds an assistant message containing a single text block. It is handy for seeding conversation history with a prior model reply.

func Complete 

func Complete(ctx context.Context, model Model, input Context, options StreamOptions) (AssistantMessage, error)

Complete runs a request on the default client and returns the final assistant message. Like Stream, it requires the model's protocol to be registered.

func NewAssistantMessage added in v0.5.2 

func NewAssistantMessage(model Model) AssistantMessage

NewAssistantMessage initializes provider-independent response metadata.

func (AssistantMessage) MarshalJSON added in v0.5.2 

func (message AssistantMessage) MarshalJSON() ([]byte, error)

func (*AssistantMessage) Text added in v0.5.2 

func (message *AssistantMessage) Text() string

Text concatenates the text from every text block in the message, in order. It ignores thinking and tool-call blocks, returning "" when there is no text.

func (*AssistantMessage) ToolCalls added in v0.5.2 

func (message *AssistantMessage) ToolCalls() []ToolCall

ToolCalls returns every tool call in the message, in order. It returns nil when the message requested no tools.

func (*AssistantMessage) UnmarshalJSON added in v0.5.2 

func (message *AssistantMessage) UnmarshalJSON(data []byte) error

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client routes LLM requests to the adapter registered for a model protocol.

func NewClient 

func NewClient(registry *Registry) *Client

NewClient creates a client backed by the given provider registry.

func (*Client) Complete added in v0.5.2 

func (c *Client) Complete(
	ctx context.Context,
	model Model,
	input Context,
	options StreamOptions,
) (AssistantMessage, error)

Complete consumes a provider stream and returns the final assistant message.

func (*Client) Stream added in v0.5.2 

func (c *Client) Stream(ctx context.Context, model Model, input Context, options StreamOptions) (<-chan Event, error)

Stream starts a streaming completion request for the given model and input.

type Context 

type Context struct {
	SystemPrompt string           `json:"systemPrompt,omitempty"`
	Messages     []Message        `json:"messages"`
	Tools        []ToolDefinition `json:"tools,omitempty"`
}

Context contains the prompt, conversation history, and available tools.

func NewContext added in v0.5.1 

func NewContext(messages ...Message) Context

NewContext assembles a Context from the given messages.

func Prompt added in v0.5.1 

func Prompt(text string) Context

Prompt builds a Context holding a single user text message. It is the shortest way to start a one-shot completion.

func PromptWithSystem added in v0.5.1 

func PromptWithSystem(system, user string) Context

PromptWithSystem builds a Context with a system prompt and a single user text message.

func (Context) MarshalJSON added in v0.5.2 

func (input Context) MarshalJSON() ([]byte, error)

func (*Context) UnmarshalJSON added in v0.5.2 

func (input *Context) UnmarshalJSON(data []byte) error

type Diagnostic added in v0.2.0 

type Diagnostic struct {
	// Type categorizes the event, e.g. DiagnosticToolArgumentsRecovered.
	Type string `json:"type"`
	// Timestamp is the Unix millisecond time the diagnostic was recorded.
	Timestamp int64 `json:"timestamp"`
	// Message is an optional human-readable summary.
	Message string `json:"message,omitempty"`
	// Details carries structured, redacted context for the event.
	Details map[string]any `json:"details,omitempty"`
}

Diagnostic records a non-fatal event that occurred while producing an assistant response — a failure that was recovered from, or a degraded result — without affecting the message content or stop reason. Callers may inspect AssistantMessage.Diagnostics to react to recoveries, for example to avoid auto-executing a tool call whose arguments were only partially recovered.

func ToolArgumentsDiagnostic added in v0.5.2 

func ToolArgumentsDiagnostic(toolCallID, toolName string, mode ArgumentsMode) (Diagnostic, bool)

ToolArgumentsDiagnostic builds a diagnostic describing how a tool call's arguments were recovered. ok is false for a clean (strict) parse, which needs no diagnostic.

type Event 

type Event struct {
	Type EventType

	ContentIndex int

	Delta string

	Content string

	ToolCall *ToolCall

	Partial *AssistantMessage

	Message *AssistantMessage

	Err error
}

Event is a single update emitted while streaming a provider response.

type EventType 

type EventType string

EventType identifies the kind of update emitted by a provider stream. 

const (
	// EventStart marks the beginning of a provider stream.
	EventStart EventType = "start"
	// EventTextStart marks the creation of a text content block.
	EventTextStart EventType = "text_start"
	// EventTextDelta carries newly generated text.
	EventTextDelta EventType = "text_delta"
	// EventTextEnd carries the completed text content block.
	EventTextEnd EventType = "text_end"
	// EventThinkingStart marks the creation of a reasoning content block.
	EventThinkingStart EventType = "thinking_start"
	// EventThinkingDelta carries newly generated reasoning content.
	EventThinkingDelta EventType = "thinking_delta"
	// EventThinkingEnd carries the completed reasoning content block.
	EventThinkingEnd EventType = "thinking_end"
	// EventToolCallStart marks the creation of a tool call content block.
	EventToolCallStart EventType = "toolcall_start"
	// EventToolCallDelta carries a fragment of a tool call's arguments as it streams.
	EventToolCallDelta EventType = "toolcall_delta"
	// EventToolCallEnd carries a tool call whose arguments finished streaming.
	// Malformed or truncated argument JSON is parsed best-effort, so callers
	// should validate the arguments and wait for EventDone before executing the
	// call, because a later content block may still fail the overall response.
	EventToolCallEnd EventType = "toolcall_end"
	// EventDone carries the final assistant message.
	EventDone EventType = "done"
	// EventError carries a stream failure.
	EventError EventType = "error"
)

type ImageContent 

type ImageContent struct {
	Data     string `json:"data"`
	MIMEType string `json:"mimeType"`
}

ImageContent represents a base64-encoded image.

func (ImageContent) MarshalJSON added in v0.5.2 

func (content ImageContent) MarshalJSON() ([]byte, error)

type Message 

type Message interface {
	// contains filtered or unexported methods
}

Message is one item in the conversation context.

func TransformMessages 

func TransformMessages(messages []Message, model Model, normalizeToolCallID func(string) string) []Message

TransformMessages prepares the library's provider-independent conversation history for replay against model. Provider adapters should call it before translating Message values into their own wire-format message types.

Transformation happens per request instead of modifying the Agent's canonical history. The same history may later be sent to a model with different image, reasoning, or tool capabilities. Mutating stored history would make that model switch lose information permanently.

Shared transformations are applied in this order:

  1. Replace images with descriptive text when the target model is text-only.
  2. Reconcile assistant turns produced by a different model: keep reasoning for the same model, downgrade it to text otherwise, and normalize tool-call identifiers via normalizeToolCallID when crossing providers.
  3. Drop assistant turns terminated by an error or cancellation because they may contain partial reasoning or half-streamed tool calls.
  4. Insert synthetic error results for tool calls with no matching result before the conversation continues or ends.

normalizeToolCallID rewrites a tool-call ID for the target provider; pass nil to leave identifiers unchanged.

The returned slice is new and this function does not mutate messages. Message objects requiring no changes may still be shared with the input, so callers should treat the input and result as immutable.

type Model 

type Model struct {
	// Identity: how the model is named and grouped.
	ID       string `json:"id"`       // stable identifier sent to the provider
	Name     string `json:"name"`     // human-readable display name
	Provider string `json:"provider"` // vendor key, e.g. "anthropic", "openai"

	// Routing: how to talk to the model. Protocol is the discriminator the
	// Client uses to pick an adapter (see Client.Stream); BaseURL and Headers
	// let a compatible vendor reuse a protocol against its own endpoint.
	Protocol Protocol          `json:"protocol"`
	BaseURL  string            `json:"baseUrl"`
	Headers  map[string]string `json:"headers,omitempty"`

	// Capabilities: what the model can do and its size limits.
	Reasoning bool `json:"reasoning"` // whether the model can produce thinking
	// ThinkingLevelMap maps a provider-independent level to the provider's own
	// value; a nil value marks a level as unsupported, a missing key falls back
	// to the provider default.
	ThinkingLevelMap map[ModelThinkingLevel]*string `json:"thinkingLevelMap,omitempty"`
	Input            []ModelInput                   `json:"input"`         // accepted modalities: text, image
	ContextWindow    int64                          `json:"contextWindow"` // max total tokens (input + output)
	MaxTokens        int64                          `json:"maxTokens"`     // max tokens the model may generate

	// Pricing and per-provider quirks.
	Cost ModelCost `json:"cost"`
	// Compatibility carries protocol-specific overrides for vendors that
	// deviate from the reference API. Its concrete type is selected at decode
	// time by Protocol (see UnmarshalJSON below).
	Compatibility ModelCompatibility `json:"compat,omitempty"`
}

Model identifies a model, its provider endpoint, capabilities, limits, and pricing. ThinkingLevelMap values are provider-specific; nil marks a level as unsupported while a missing key uses the provider default.

func GetModel 

func GetModel(provider, modelID string) Model

GetModel returns a model from the package's built-in model registry. It panics when the provider/model pair is unknown. Use LookupModel when the identifiers come from dynamic or untrusted input.

func GetModels 

func GetModels(provider string) []Model

GetModels returns all built-in models for a provider.

func LookupModel 

func LookupModel(provider, modelID string) (Model, bool)

LookupModel returns a model from the package's built-in model registry.

func (*Model) UnmarshalJSON added in v0.5.2 

func (model *Model) UnmarshalJSON(data []byte) error

UnmarshalJSON restores the concrete compatibility type selected by Protocol. The protocol acts as the discriminator, selecting the concrete per-protocol compatibility type at runtime.

type ModelCompatibility 

type ModelCompatibility interface {
	Protocol() Protocol
}

ModelCompatibility is implemented by protocol-specific compatibility configurations. It keeps Model independent from any one provider protocol while allowing registration and adapters to verify type/protocol agreement.

type ModelCost 

type ModelCost struct {
	Input      float64 `json:"input"`
	Output     float64 `json:"output"`
	CacheRead  float64 `json:"cacheRead"`
	CacheWrite float64 `json:"cacheWrite"`
}

ModelCost stores prices in US dollars per million tokens.

type ModelInput 

type ModelInput string

ModelInput identifies an input modality accepted by a model. 

const (
	Text  ModelInput = "text"
	Image ModelInput = "image"
)

type ModelRegistry added in v0.5.2 

type ModelRegistry struct {
	// contains filtered or unexported fields
}

ModelRegistry stores models by provider and model ID. It is safe for concurrent access and returns defensive copies of registered models.

func NewModelRegistry added in v0.5.2 

func NewModelRegistry() *ModelRegistry

NewModelRegistry creates an empty model registry.

func (*ModelRegistry) Get added in v0.5.2 

func (registry *ModelRegistry) Get(provider, modelID string) (Model, bool)

Get returns a model registered for provider and modelID.

func (*ModelRegistry) Models added in v0.5.2 

func (registry *ModelRegistry) Models(provider string) []Model

Models returns a provider's models ordered by model ID.

func (*ModelRegistry) Providers added in v0.5.2 

func (registry *ModelRegistry) Providers() []string

Providers returns registered provider IDs in lexical order.

func (*ModelRegistry) Register added in v0.5.2 

func (registry *ModelRegistry) Register(model Model) error

Register adds or replaces a model with the same provider and ID.

type ModelThinkingLevel 

type ModelThinkingLevel string

ModelThinkingLevel is a provider-independent reasoning effort level. 

const (
	ModelThinkingOff     ModelThinkingLevel = "off"
	ModelThinkingMinimal ModelThinkingLevel = "minimal"
	ModelThinkingLow     ModelThinkingLevel = "low"
	ModelThinkingMedium  ModelThinkingLevel = "medium"
	ModelThinkingHigh    ModelThinkingLevel = "high"
	ModelThinkingXHigh   ModelThinkingLevel = "xhigh"
)

func ClampThinkingLevel 

func ClampThinkingLevel(model Model, level ModelThinkingLevel) ModelThinkingLevel

ClampThinkingLevel adjusts a requested level to the nearest one the model supports: it prefers the requested level, then steps up, then down, and falls back to the lowest supported level (or "off").

func SupportedThinkingLevels 

func SupportedThinkingLevels(model Model) []ModelThinkingLevel

SupportedThinkingLevels returns the thinking levels a model accepts. A non-reasoning model supports only "off". For reasoning models, a level mapped to nil is unsupported, and "xhigh" is supported only when explicitly mapped.

type OpenAICompletionsCompatibility 

type OpenAICompletionsCompatibility struct {
	SupportsStore                               *bool  `json:"supportsStore,omitempty"`
	SupportsDeveloperRole                       *bool  `json:"supportsDeveloperRole,omitempty"`
	SupportsReasoningEffort                     *bool  `json:"supportsReasoningEffort,omitempty"`
	MaxTokensField                              string `json:"maxTokensField,omitempty"`
	SupportsStrictMode                          *bool  `json:"supportsStrictMode,omitempty"`
	RequiresReasoningContentOnAssistantMessages *bool  `json:"requiresReasoningContentOnAssistantMessages,omitempty"`
	// RequiresThinkingAsText makes replayed assistant turns carry thinking as a
	// leading text content block instead of a provider reasoning field, for
	// endpoints that reject reasoning fields on input.
	RequiresThinkingAsText *bool  `json:"requiresThinkingAsText,omitempty"`
	ThinkingFormat         string `json:"thinkingFormat,omitempty"`
	ZAIToolStream          *bool  `json:"zaiToolStream,omitempty"`
}

OpenAICompletionsCompatibility describes differences between providers that implement an OpenAI-compatible Chat Completions endpoint. Pointer booleans distinguish an explicit false value from an unspecified provider default.

func (*OpenAICompletionsCompatibility) Protocol added in v0.5.2 

func (*OpenAICompletionsCompatibility) Protocol() Protocol

Protocol identifies the API protocol whose request and message dialect this compatibility configuration describes.

type OpenAICompletionsStreamOptions added in v0.2.0 

type OpenAICompletionsStreamOptions struct {
	ToolChoice OpenAIToolChoice
}

OpenAICompletionsStreamOptions contains settings understood only by the OpenAI-compatible Chat Completions protocol.

func (*OpenAICompletionsStreamOptions) Protocol added in v0.5.2 

func (*OpenAICompletionsStreamOptions) Protocol() Protocol

Protocol identifies the protocol that accepts these options.

func (*OpenAICompletionsStreamOptions) Validate added in v0.5.2 

func (options *OpenAICompletionsStreamOptions) Validate(tools []ToolDefinition) error

Validate checks the OpenAI-specific settings against the available tools.

type OpenAIToolChoice added in v0.2.0 

type OpenAIToolChoice interface {
	// contains filtered or unexported methods
}

OpenAIToolChoice is the native OpenAI Chat Completions tool_choice union. Use one of the OpenAIToolChoice* mode constants or OpenAIToolChoiceFunction.

type OpenAIToolChoiceFunction added in v0.2.0 

type OpenAIToolChoiceFunction struct {
	Name string
}

OpenAIToolChoiceFunction forces the model to call the named function tool.

type OpenAIToolChoiceMode added in v0.2.0 

type OpenAIToolChoiceMode string

OpenAIToolChoiceMode is one of OpenAI's string tool-choice modes. 

const (
	OpenAIToolChoiceAuto     OpenAIToolChoiceMode = "auto"
	OpenAIToolChoiceNone     OpenAIToolChoiceMode = "none"
	OpenAIToolChoiceRequired OpenAIToolChoiceMode = "required"
)

type Protocol 

type Protocol string

Protocol identifies the API protocol used to communicate with a model. 

const (
	ProtocolOpenAICompletions Protocol = "openai-completions"
	ProtocolAnthropicMessages Protocol = "anthropic-messages"
)

type ProtocolAdapter added in v0.2.0 

type ProtocolAdapter interface {
	// Protocol returns the registry key used to select this adapter.
	Protocol() Protocol

	// Stream emits response events for the given model and conversation context.
	Stream(ctx context.Context, model Model, input Context, options StreamOptions) (<-chan Event, error)
}

ProtocolAdapter translates between a concrete LLM protocol and the package streaming interface.

type ProtocolStreamOptions added in v0.2.0 

type ProtocolStreamOptions interface {
	Protocol() Protocol
	Validate(tools []ToolDefinition) error
}

ProtocolStreamOptions is the extension point for settings whose semantics cannot be shared across protocols. Custom protocol adapters may provide their own implementation and validate it before the stream starts.

type ProviderEnv 

type ProviderEnv map[string]string

ProviderEnv contains request-scoped environment overrides. Non-empty values take precedence over process environment variables during credential lookup.

type Registry added in v0.2.0 

type Registry struct {
	// contains filtered or unexported fields
}

Registry stores protocol adapters and is safe for concurrent access.

func NewRegistry added in v0.2.0 

func NewRegistry() *Registry

NewRegistry creates an empty provider registry.

func (*Registry) Get added in v0.5.2 

func (registry *Registry) Get(protocol Protocol) (ProtocolAdapter, bool)

Get returns the adapter registered for the protocol.

func (*Registry) Register added in v0.5.2 

func (registry *Registry) Register(adapter ProtocolAdapter) error

Register adds or replaces an adapter for its protocol.

type StopReason 

type StopReason string

StopReason explains why the model stopped generating a response. 

const (
	// StopReasonStop marks a normal completion.
	StopReasonStop StopReason = "stop"
	// StopReasonLength marks truncation by the max output token limit.
	StopReasonLength StopReason = "length"
	// StopReasonToolUse marks a stop to let the caller execute tool calls.
	StopReasonToolUse StopReason = "toolUse"
	// StopReasonError marks a provider or runtime failure.
	StopReasonError StopReason = "error"
	// StopReasonAborted marks a cancelled request.
	StopReasonAborted StopReason = "aborted"
)

type StreamOptions 

type StreamOptions struct {
	APIKey string
	Env    ProviderEnv
	// Temperature overrides the model's default sampling temperature when set.
	Temperature *float64
	// MaxTokens caps the output tokens for this request. Zero leaves it unset.
	MaxTokens int64
	// Headers are merged into the request, overriding model default headers.
	Headers map[string]string
	// Reasoning requests a thinking level. The provider clamps it to what the
	// model supports. Empty leaves the model's default; "off" disables thinking.
	Reasoning ModelThinkingLevel
	// ProtocolOptions carries settings specific to exactly one protocol.
	ProtocolOptions ProtocolStreamOptions
	// MaxRetries overrides the SDK client-side retry count for transient failures
	// (HTTP 429 and 5xx, connection errors). Nil leaves the SDK default; a value
	// of 0 disables retries so the caller can manage them.
	MaxRetries *int
	// Timeout caps the total duration of one HTTP attempt. Zero leaves the SDK
	// default. It is independent of the request context, which still cancels the
	// whole call.
	Timeout time.Duration
	// OnResponse, when set, is called with the status and headers of every HTTP
	// response before its body is consumed. It fires once per attempt, so a
	// retried request invokes it for each try, making retries observable.
	OnResponse func(status int, headers http.Header)
	// OnRequest, when set, is called with the method, URL, and full body of every
	// HTTP request before it is sent. The body is the exact JSON serialized for
	// the provider. It fires once per attempt, so a retried request invokes it
	// for each try, making retries observable.
	OnRequest func(method, url string, body []byte)
	// RewriteRequest, when set, transforms the serialized request body before it
	// is sent. It receives the method, URL, and body, and returns the body to
	// send; returning nil leaves the body unchanged. Use it to patch
	// provider-specific fields the typed API does not expose. It fires once per
	// attempt and always rewrites the original body, so a retried request is
	// rewritten consistently.
	RewriteRequest func(method, url string, body []byte) []byte
}

StreamOptions contains shared request settings plus optional protocol-specific extensions. A non-nil extension must match the target model protocol.

func (StreamOptions) Validate added in v0.5.2 

func (options StreamOptions) Validate(protocol Protocol, tools []ToolDefinition) error

Validate checks that explicitly supplied protocol extensions match the target protocol and contain supported values.

type StreamWriter added in v0.2.0 

type StreamWriter struct {
	// contains filtered or unexported fields
}

StreamWriter manages the event channel for one streamed response. It emits a single EventStart, attaches a Partial snapshot to every non-terminal event, and guarantees exactly one terminal event (EventDone or EventError) before the channel closes. A cancelled context is reported as StopReasonAborted.

A protocol adapter builds its AssistantMessage incrementally — the writer is constructed over a pointer to it — emits deltas through Emit, and ends with Done on success or Fail on error. This is the shared machinery behind the built-in adapters; it keeps the single-terminal guarantee and Partial cloning in one place rather than duplicated per provider.

func NewStreamWriter added in v0.2.0 

func NewStreamWriter(ctx context.Context, events chan<- Event, output *AssistantMessage) *StreamWriter

NewStreamWriter returns a writer that sends events to the channel and snapshots output for each event's Partial and for the terminal Message.

func (*StreamWriter) Done added in v0.5.2 

func (w *StreamWriter) Done()

Done emits the single terminal EventDone carrying the final message.

func (*StreamWriter) Emit added in v0.5.2 

func (w *StreamWriter) Emit(event Event)

Emit sends a non-terminal event, first emitting EventStart if needed and attaching a fresh Partial snapshot of the message built so far.

func (*StreamWriter) Fail added in v0.5.2 

func (w *StreamWriter) Fail(err error)

Fail emits the single terminal EventError. A cancelled context is reported as StopReasonAborted and replaces the error with the context error; any other failure is reported as StopReasonError.

func (*StreamWriter) Start added in v0.5.2 

func (w *StreamWriter) Start()

Start emits EventStart. It is idempotent: only the first call emits.

type TextContent 

type TextContent struct {
	Text          string `json:"text"`
	TextSignature string `json:"textSignature,omitempty"`
}

TextContent represents plain text.

func (TextContent) MarshalJSON added in v0.5.2 

func (content TextContent) MarshalJSON() ([]byte, error)

type ThinkingContent 

type ThinkingContent struct {
	Thinking          string `json:"thinking"`
	ThinkingSignature string `json:"thinkingSignature,omitempty"`
	Redacted          bool   `json:"redacted,omitempty"`
}

ThinkingContent represents model reasoning content.

func (ThinkingContent) MarshalJSON added in v0.5.2 

func (content ThinkingContent) MarshalJSON() ([]byte, error)

type ThinkingDisplay added in v0.2.0 

type ThinkingDisplay string

ThinkingDisplay controls how a reasoning model returns its thinking. It does not change whether the model reasons or what it is billed; it only governs what travels back. Only Anthropic-protocol models honor it today. 

const (
	// ThinkingDisplaySummarized returns summarized thinking text in the response.
	ThinkingDisplaySummarized ThinkingDisplay = "summarized"
	// ThinkingDisplayOmitted redacts the thinking text but still returns the
	// signature needed for multi-turn tool-use continuity. Use it for backends
	// that never surface reasoning, trading the thinking text for lower
	// time-to-first-token and a lighter response.
	ThinkingDisplayOmitted ThinkingDisplay = "omitted"
)

type ToolCall 

type ToolCall struct {
	ID               string         `json:"id"`
	Name             string         `json:"name"`
	Arguments        map[string]any `json:"arguments"`
	ThoughtSignature string         `json:"thoughtSignature,omitempty"`
}

ToolCall describes a request to invoke a named tool with JSON arguments.

func CloneToolCall added in v0.2.0 

func CloneToolCall(toolCall *ToolCall) *ToolCall

CloneToolCall returns a deep copy of a tool call, including its arguments, for use in an event's ToolCall field.

func (ToolCall) MarshalJSON added in v0.5.2 

func (content ToolCall) MarshalJSON() ([]byte, error)

type ToolDefinition 

type ToolDefinition struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters"`
}

ToolDefinition describes a tool that the model may call.

func MustTool 

func MustTool[T any](name, description string) ToolDefinition

MustTool is NewTool for statically declared tools. It panics when the tool name or argument type cannot produce a valid definition.

func NewTool 

func NewTool[T any](name, description string) (ToolDefinition, error)

NewTool creates a tool definition whose parameters are generated from T. T must be a struct or pointer to a struct. Fields without json omitempty are required, and jsonschema tags can add descriptions, enums, and constraints.

type ToolResultContent 

type ToolResultContent interface {
	// contains filtered or unexported methods
}

ToolResultContent is content that can appear in a tool result message.

type ToolResultMessage 

type ToolResultMessage struct {
	ToolCallID string              `json:"toolCallId"`
	ToolName   string              `json:"toolName"`
	Content    []ToolResultContent `json:"content"`
	IsError    bool                `json:"isError"`
}

ToolResultMessage contains the result of an assistant tool call.

func ToolResult added in v0.5.1 

func ToolResult(callID, toolName, text string) *ToolResultMessage

ToolResult builds a tool result message answering the assistant tool call with the given ID. The text becomes a single text block in the result.

func (ToolResultMessage) MarshalJSON added in v0.5.2 

func (message ToolResultMessage) MarshalJSON() ([]byte, error)

func (*ToolResultMessage) UnmarshalJSON added in v0.5.2 

func (message *ToolResultMessage) UnmarshalJSON(data []byte) error

type Usage 

type Usage struct {
	Input       int64     `json:"input"`
	Output      int64     `json:"output"`
	CacheRead   int64     `json:"cacheRead"`
	CacheWrite  int64     `json:"cacheWrite"`
	TotalTokens int64     `json:"totalTokens"`
	Cost        UsageCost `json:"cost"`
}

Usage records token consumption for one assistant response.

type UsageCost 

type UsageCost struct {
	Input      float64 `json:"input"`
	Output     float64 `json:"output"`
	CacheRead  float64 `json:"cacheRead"`
	CacheWrite float64 `json:"cacheWrite"`
	Total      float64 `json:"total"`
}

UsageCost breaks down the US dollar cost of one response by token category.

func CalculateCost 

func CalculateCost(model Model, usage Usage) UsageCost

CalculateCost returns the US dollar cost of usage at the model's prices. Model costs are quoted per million tokens.

type UserContent 

type UserContent interface {
	// contains filtered or unexported methods
}

UserContent is content that can appear in a user message.

type UserMessage 

type UserMessage struct {
	Content []UserContent `json:"content"`
}

UserMessage contains content supplied by the user.

func UserImage added in v0.5.1 

func UserImage(data, mimeType string) *UserMessage

UserImage builds a user message containing a single base64-encoded image.

func UserText added in v0.5.1 

func UserText(text string) *UserMessage

UserText builds a user message containing a single text block.

func (UserMessage) MarshalJSON added in v0.5.2 

func (message UserMessage) MarshalJSON() ([]byte, error)

func (*UserMessage) UnmarshalJSON added in v0.5.2 

func (message *UserMessage) UnmarshalJSON(data []byte) error

Source Files

View all Source files

Directories

Path Synopsis
Package all registers every built-in protocol adapter into the llm package default registry.
 Package all registers every built-in protocol adapter into the llm package default registry.
Package anthropic implements the Anthropic Messages protocol on top of the official anthropic-sdk-go.
 Package anthropic implements the Anthropic Messages protocol on top of the official anthropic-sdk-go.
internal
genmodels command
Command genmodels builds llm's checked-in model catalog from public model catalogs.
 Command genmodels builds llm's checked-in model catalog from public model catalogs.
jsonx
Package jsonx provides best-effort JSON recovery for model output.
 Package jsonx provides best-effort JSON recovery for model output.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.