llm

Documentation

Overview

Package llm defines provider-neutral request, response, streaming, and error DTOs.

The package intentionally contains no provider wire formats and no librecode runtime, database, tool, model, or extension imports. Provider packages should translate between their HTTP APIs and these types; assistant orchestration should translate between persisted session state and these types.

This package is the provider boundary for generation requests and responses. Assistant code translates persisted session state into these DTOs, while provider code translates them into HTTP payloads and normalized responses.

Index

• func IsKind(err error, kind ErrorKind) bool
• type Auth
• type CompletedRound
• type ErrorKind
• type FinishReason
• type Generator
• type HookInput
• type HookOutput
• type Message
  • func TextMessage(role Role, text string) Message
• type ModelRef
• type Part
  • func TextPart(text string) Part
• type PartType
• type ProviderError
  • func AsProviderError(err error) (*ProviderError, bool)
  • func (err *ProviderError) Error() string
  • func (err *ProviderError) Unwrap() error
• type ProviderObserver
• type ProviderRequestHook
• type ProviderResponseObserver
• type Request
  • func EmptyRequest() Request
• type Response
• type Role
• type RoundCheckpoint
• type Stream
• type StreamChunk
• type Streamer
• type TerminationMetadata
  • func NewTerminationMetadata(status, finish, incomplete string) TerminationMetadata
• type TokenContributor
  • func CloneTokenContributors(contributors []TokenContributor) []TokenContributor
• type ToolCall
• type ToolDefinition
• type ToolExecutor
• type ToolResult
• type Usage
  • func EmptyUsage() Usage
  • func MergeUsage(estimated, reported *Usage) Usage
  • func (usage *Usage) ContextPercent() int
  • func (usage *Usage) HasAny() bool
  • func (usage *Usage) Reported() bool
  • func (usage *Usage) TotalTokens() int
  • func (usage *Usage) WithReported() Usage
• type UsageProvenance

Functions

func IsKind

func IsKind(err error, kind ErrorKind) bool

IsKind reports whether err has the given provider error kind.

Types

type Auth

type Auth struct {
	Headers map[string]string `json:"headers,omitempty"`
	APIKey  string            `json:"api_key,omitempty"`
}

Auth contains provider request credentials and extra headers.

type CompletedRound

type CompletedRound struct {
	Assistant    Message      `json:"assistant"`
	ToolResults  []ToolResult `json:"tool_results,omitempty"`
	FinishReason FinishReason `json:"finish_reason,omitempty"`
	Usage        Usage        `json:"usage"`
}

CompletedRound is the provider-neutral artifact produced by one model turn. Usage contains only this provider response, not cumulative completion usage.

type ErrorKind

type ErrorKind string

ErrorKind identifies provider error classes that assistant orchestration can handle.

const (
	// ErrorKindUnknown is an unclassified provider error.
	ErrorKindUnknown ErrorKind = "unknown"
	// ErrorKindAuth is an authentication or authorization error.
	ErrorKindAuth ErrorKind = "auth"
	// ErrorKindRateLimit is a provider rate-limit error.
	ErrorKindRateLimit ErrorKind = "rate_limit"
	// ErrorKindContextOverflow means the provider rejected the request as too large.
	ErrorKindContextOverflow ErrorKind = "context_overflow"
	// ErrorKindTimeout is a provider timeout.
	ErrorKindTimeout ErrorKind = "timeout"
	// ErrorKindNetwork is a transport error before a provider response was received.
	ErrorKindNetwork ErrorKind = "network"
	// ErrorKindDecode is an invalid or unsupported provider response shape.
	ErrorKindDecode ErrorKind = "decode"
	// ErrorKindServer is a provider server-side error.
	ErrorKindServer ErrorKind = "server"
	// ErrorKindBadRequest is a provider-side request validation error.
	ErrorKindBadRequest ErrorKind = "bad_request"
)

type FinishReason

type FinishReason string

FinishReason describes why a model response stopped.

const (
	// FinishReasonUnknown means the provider did not report a stop reason.
	FinishReasonUnknown FinishReason = ""
	// FinishReasonStop means the model completed normally.
	FinishReasonStop FinishReason = "stop"
	// FinishReasonLength means the model hit an output or context limit.
	FinishReasonLength FinishReason = "length"
	// FinishReasonToolCalls means the model stopped to request tool execution.
	FinishReasonToolCalls FinishReason = "tool-calls"
	// FinishReasonContentFilter means provider policy filtered the response.
	FinishReasonContentFilter FinishReason = "content-filter"
	// FinishReasonRefusal means the model declined the request as a successful response.
	FinishReasonRefusal FinishReason = "refusal"
	// FinishReasonError means the provider reported a generation error.
	FinishReasonError FinishReason = "error"
	// FinishReasonAborted means generation was canceled or aborted.
	FinishReasonAborted FinishReason = "aborted"
)

type Generator

type Generator interface {
	Generate(ctx context.Context, request *Request) (*Response, error)
}

Generator generates model responses from provider-neutral requests.

type HookInput

type HookInput struct {
	ProviderOptions map[string]any    `json:"provider_options,omitempty"`
	Payload         map[string]any    `json:"payload,omitempty"`
	Headers         map[string]string `json:"headers,omitempty"`
	SessionID       string            `json:"session_id,omitempty"`
	ThinkingLevel   string            `json:"thinking_level,omitempty"`
	Model           ModelRef          `json:"model"`
	MaxTokens       int               `json:"max_tokens,omitempty"`
	Attempt         int               `json:"attempt"`
}

HookInput describes a provider wire request before it is sent.

type HookOutput

type HookOutput struct {
	Payload map[string]any    `json:"payload,omitempty"`
	Headers map[string]string `json:"headers,omitempty"`
}

HookOutput describes a provider wire request after hook mutation.

type Message

type Message struct {
	Metadata map[string]any `json:"metadata,omitempty"`
	Role     Role           `json:"role"`
	Content  []Part         `json:"content,omitempty"`
}

Message is one provider-neutral model-facing message.

func TextMessage

func TextMessage(role Role, text string) Message

TextMessage creates a message with one text part.

type ModelRef

type ModelRef struct {
	Metadata         map[string]any     `json:"metadata,omitempty"`
	ThinkingLevelMap map[string]*string `json:"thinking_level_map,omitempty"`
	Provider         string             `json:"provider"`
	ID               string             `json:"id"`
	API              string             `json:"api,omitempty"`
	BaseURL          string             `json:"base_url,omitempty"`
	MaxTokens        int                `json:"max_tokens,omitempty"`
	ContextWindow    int                `json:"context_window,omitempty"`
	Reasoning        bool               `json:"reasoning,omitempty"`
}

ModelRef identifies the concrete model and provider endpoint family.

type Part

type Part struct {
	Metadata   map[string]any `json:"metadata,omitempty"`
	ToolCall   *ToolCall      `json:"tool_call,omitempty"`
	ToolResult *ToolResult    `json:"tool_result,omitempty"`
	Type       PartType       `json:"type"`
	Text       string         `json:"text,omitempty"`
	Data       string         `json:"data,omitempty"`
	MIMEType   string         `json:"mime_type,omitempty"`
}

Part is one typed content block inside a message or response. Image/document inputs and provider cache-control metadata are intentionally flattened for this staged boundary and can grow when runtime support lands.

func TextPart

func TextPart(text string) Part

TextPart creates one text part.

type PartType

type PartType string

PartType identifies a typed content block.

const (
	// PartText is ordinary natural-language text.
	PartText PartType = "text"
	// PartReasoning is provider reasoning or thinking content.
	PartReasoning PartType = "reasoning"
	// PartImage is an inline image content block.
	PartImage PartType = "image"
	// PartFile is a file/document content block.
	PartFile PartType = "file"
	// PartSource is a source citation content block.
	PartSource PartType = "source"
	// PartToolCall is a tool-call content block.
	PartToolCall PartType = "tool_call"
	// PartToolResult is a tool-result content block.
	PartToolResult PartType = "tool_result"
)

type ProviderError

type ProviderError struct {
	Cause        error          `json:"-"`
	Metadata     map[string]any `json:"metadata,omitempty"`
	Kind         ErrorKind      `json:"kind"`
	Provider     string         `json:"provider,omitempty"`
	Model        string         `json:"model,omitempty"`
	Code         string         `json:"code,omitempty"`
	ProviderCode string         `json:"provider_code,omitempty"`
	Message      string         `json:"message"`
	StatusCode   int            `json:"status_code,omitempty"`
}

ProviderError is a typed provider error with optional provider metadata.

func AsProviderError

func AsProviderError(err error) (*ProviderError, bool)

AsProviderError returns err as a ProviderError when possible.

func (*ProviderError) Error

func (err *ProviderError) Error() string

Error returns a human-readable provider error message.

func (*ProviderError) Unwrap

func (err *ProviderError) Unwrap() error

Unwrap returns the wrapped cause.

type ProviderObserver

type ProviderObserver func(context.Context, *HookInput)

ProviderObserver observes provider attempts without mutating them.

type ProviderRequestHook

type ProviderRequestHook func(context.Context, *HookInput) (HookOutput, error)

ProviderRequestHook can inspect and conservatively mutate a provider wire request. It returns HookOutput by value so a hook cannot return a nil output object.

type ProviderResponseObserver

type ProviderResponseObserver func(context.Context, Usage)

ProviderResponseObserver observes usage from one successfully parsed provider response.

type Request

type Request struct {
	ProviderOptions map[string]any   `json:"provider_options,omitempty"`
	Auth            Auth             `json:"auth"`
	SystemPrompt    string           `json:"system_prompt,omitempty"`
	ThinkingLevel   string           `json:"thinking_level,omitempty"`
	SessionID       string           `json:"session_id,omitempty"`
	Messages        []Message        `json:"messages"`
	Tools           []ToolDefinition `json:"tools,omitempty"`
	Model           ModelRef         `json:"model"`
	Usage           Usage            `json:"usage"`
	MaxTokens       int              `json:"max_tokens,omitempty"`
	DisableTools    bool             `json:"disable_tools,omitempty"`
}

Request describes one provider-neutral LLM generation call.

func EmptyRequest

func EmptyRequest() Request

EmptyRequest returns an explicit zero-value provider-neutral request.

type Response

type Response struct {
	FinishReason FinishReason        `json:"finish_reason,omitempty"`
	Termination  TerminationMetadata `json:"termination"`
	Content      []Part              `json:"content,omitempty"`
	ToolCalls    []ToolCall          `json:"tool_calls,omitempty"`
	Usage        Usage               `json:"usage"`
}

Response is a completed provider-neutral LLM response.

type Role

type Role string

Role identifies the speaker for a model-facing message.

const (
	// RoleSystem is provider or application instruction text.
	RoleSystem Role = "system"
	// RoleUser is user-authored text or app-authored context shown to the model.
	RoleUser Role = "user"
	// RoleAssistant is assistant-authored output.
	RoleAssistant Role = "assistant"
	// RoleTool is a tool result message.
	RoleTool Role = "tool"
)

type RoundCheckpoint

type RoundCheckpoint func(context.Context, *CompletedRound) ([]Message, error)

RoundCheckpoint runs after one provider response and its complete tool batch settle. Returned user messages are appended before the next provider request. An empty result lets a final response return or an existing tool continuation proceed.

type Stream

type Stream interface {
	Recv() (*StreamChunk, error)
	Close() error
}

Stream yields provider-neutral response chunks.

type StreamChunk

type StreamChunk struct {
	Part         *Part        `json:"part,omitempty"`
	ToolCall     *ToolCall    `json:"tool_call,omitempty"`
	FinishReason FinishReason `json:"finish_reason,omitempty"`
	Usage        Usage        `json:"usage"`
}

StreamChunk is one provider-neutral streaming delta.

type Streamer

type Streamer interface {
	Stream(ctx context.Context, request Request) (Stream, error)
}

Streamer generates streamed model responses from provider-neutral requests.

type TerminationMetadata

type TerminationMetadata struct {
	ProviderStatus       string `json:"provider_status,omitempty"`
	ProviderFinishReason string `json:"provider_finish_reason,omitempty"`
	IncompleteReason     string `json:"incomplete_reason,omitempty"`
}

TerminationMetadata preserves bounded provider enums needed to disambiguate normalized finish reasons. It never contains response text or identifiers.

func NewTerminationMetadata

func NewTerminationMetadata(status, finish, incomplete string) TerminationMetadata

NewTerminationMetadata normalizes and bounds provider-owned enum values.

type TokenContributor

type TokenContributor struct {
	Label   string `json:"label"`
	Role    string `json:"role,omitempty"`
	Preview string `json:"preview,omitempty"`
	Tokens  int    `json:"tokens"`
	Chars   int    `json:"chars"`
}

TokenContributor describes a large piece of model-facing context.

func CloneTokenContributors

func CloneTokenContributors(contributors []TokenContributor) []TokenContributor

CloneTokenContributors copies token contributor slices, preserving nil/empty-as-nil semantics.

type ToolCall

type ToolCall struct {
	Metadata      map[string]any `json:"metadata,omitempty"`
	ArgumentsJSON string         `json:"arguments_json,omitempty"`
	ID            string         `json:"id"`
	Name          string         `json:"name"`
	Arguments     tool.Arguments `json:"arguments,omitzero"`
}

ToolCall is a provider-neutral request to invoke a tool.

type ToolDefinition

type ToolDefinition struct {
	Name        string      `json:"name"`
	Description string      `json:"description"`
	Schema      tool.Schema `json:"schema"`
	ReadOnly    bool        `json:"read_only,omitempty"`
}

ToolDefinition describes a callable model tool.

type ToolExecutor

type ToolExecutor func(context.Context, []ToolCall, func(*StreamChunk)) ([]ToolResult, error)

ToolExecutor executes model-requested tool calls outside provider wire clients.

type ToolResult

type ToolResult struct {
	Metadata      map[string]any `json:"metadata,omitempty"`
	ToolCallID    string         `json:"tool_call_id"`
	ArgumentsJSON string         `json:"arguments_json,omitempty"`
	Name          string         `json:"name,omitempty"`
	Error         string         `json:"error,omitempty"`
	Content       []Part         `json:"content,omitempty"`
	IsError       bool           `json:"is_error,omitempty"`
}

ToolResult is a provider-neutral result for a tool invocation.

type Usage

type Usage struct {
	Breakdown       map[string]int     `json:"breakdown,omitempty"`
	Provenance      UsageProvenance    `json:"provenance,omitempty"`
	TopContributors []TokenContributor `json:"top_contributors,omitempty"`
	ContextWindow   int                `json:"context_window,omitempty"`
	ContextTokens   int                `json:"context_tokens,omitempty"`
	InputTokens     int                `json:"input_tokens,omitempty"`
	OutputTokens    int                `json:"output_tokens,omitempty"`
}

Usage tracks model context and request/response token counts. InputTokens and OutputTokens are cumulative across provider rounds in one completion. ContextTokens is the input size of the latest provider request.

func EmptyUsage

func EmptyUsage() Usage

EmptyUsage returns explicit zero usage.

func MergeUsage

func MergeUsage(estimated, reported *Usage) Usage

MergeUsage overlays provider-reported usage on an estimated usage snapshot.

func (*Usage) ContextPercent

func (usage *Usage) ContextPercent() int

ContextPercent returns the context-window usage percentage, if known.

func (*Usage) HasAny

func (usage *Usage) HasAny() bool

HasAny reports whether any usage field is populated.

func (*Usage) Reported

func (usage *Usage) Reported() bool

Reported reports whether the provider supplied a usage object.

func (*Usage) TotalTokens

func (usage *Usage) TotalTokens() int

TotalTokens returns input plus output tokens reported for the turn.

func (*Usage) WithReported

func (usage *Usage) WithReported() Usage

WithReported marks usage as explicitly reported by a provider, including a zero-token report.

type UsageProvenance

type UsageProvenance string

UsageProvenance identifies how context usage was calculated.

const (
	// UsageProviderReported identifies usage reported directly by a provider.
	UsageProviderReported UsageProvenance = "provider_reported"
	// UsageProviderAnchorEstimate identifies usage based on a provider anchor plus local estimates.
	UsageProviderAnchorEstimate UsageProvenance = "provider_anchor_plus_estimate"
	// UsageLocalEstimate identifies usage estimated entirely locally.
	UsageLocalEstimate UsageProvenance = "local_estimate"
)