llm

Documentation

Overview

Package llm is the agent's provider-agnostic language-model port: the single interface every model backend implements, and the neutral message/tool vocabulary the agent reasons in. It is deliberately not tied to any vendor.

The shape is the proven request/response of a tool-using chat model (a system prompt, a running list of messages, a set of callable tools, and a response that is either text or a batch of tool calls), but expressed in this package's own types. A backend adapts those types to its wire format: a direct HTTP/SDK client for a hosted model, a subprocess driving an agent CLI, or a local model. None of that leaks here, so the conversation loop, the agent runtime, and tests depend only on this port and a backend is swapped without touching them.

Like state, spine, jobs, and bus, this is a port with a zero-dependency default for tests (a deterministic scripted Model, see fake.go) and real backends as out-of-tree adapters held to the same contract.

Index

• func RetryClass(status int, quotaExhausted bool) fault.Class
• func SafeBaseURL(raw string) bool
• func SupportedImageMediaType(mediaType string) bool
• type Block
  • func ImageBlock(mediaType string, data []byte) Block
• type BlockKind
• type CacheHint
• type Image
• type Message
  • func Text(role Role, text string) Message
  • func (m Message) TextContent() string
  • func (m Message) ToolUses() []ToolUse
• type Model
• type Request
• type Response
• type Role
• type Sampling
  • func (s Sampling) Deterministic() bool
  • func (s Sampling) Normalized() Sampling
• type StopReason
• type Tool
• type ToolResult
• type ToolUse
• type Usage

Functions

func RetryClass

func RetryClass(status int, quotaExhausted bool) fault.Class

RetryClass classifies an HTTP status from a model API into a fault.Class that tells the caller whether to retry. 5xx (a server fault) and 429 (rate limited) are transient, so a worker backs off and retries. The exception that matters: a 429 that signals an exhausted quota or a billing problem is permanent, not a "slow down", so it is terminal and must fail fast rather than retry for hours against an account that cannot succeed. quotaExhausted carries the provider's own signal of that case (its error type, code, or message), since the HTTP status alone cannot distinguish a rate limit from an empty wallet. Every other status (the 4xx client errors: bad request, auth, not found) is terminal.

func SafeBaseURL

func SafeBaseURL(raw string) bool

SafeBaseURL reports whether a base URL is safe to send a credential to. A model request carries the API key in a header, so the transport must be encrypted: the URL must be https, unless it targets the loopback host (a local model server or gateway), where plaintext http is allowed because the traffic never leaves the machine. An empty string means "use the provider default" and is reported safe. Backends and the provider resolver use this to refuse a plaintext remote endpoint, so a credential is never sent where it could be sniffed in transit.

func SupportedImageMediaType

func SupportedImageMediaType(mediaType string) bool

SupportedImageMediaType reports whether the media type is one every vision backend this port targets accepts, so a caller can reject an unusable image at the point of capture rather than at the model call. The set is the common intersection: PNG, JPEG, GIF, and WebP.

Types

type Block

type Block struct {
	Kind       BlockKind       `json:"kind"`
	Text       string          `json:"text,omitempty"`
	ToolUse    *ToolUse        `json:"toolUse,omitempty"`
	ToolResult *ToolResult     `json:"toolResult,omitempty"`
	Image      *Image          `json:"image,omitempty"`
	Raw        json.RawMessage `json:"raw,omitempty"` // provider-verbatim payload for KindOpaque
}

Block is one piece of a message. Exactly one of Text, ToolUse, or ToolResult is meaningful, selected by Kind, so a message is an ordered mix of prose and tool interaction rather than a single string.

func ImageBlock

func ImageBlock(mediaType string, data []byte) Block

ImageBlock builds an image content block from encoded bytes and their media type, for composing into a user message alongside text blocks.

type BlockKind

type BlockKind string

BlockKind is the type of one content block within a message.

const (
	// KindText is a run of natural-language text.
	KindText BlockKind = "text"
	// KindToolUse is the model asking to call a tool, with arguments.
	KindToolUse BlockKind = "tool_use"
	// KindToolResult is the outcome of a tool call, fed back to the model.
	KindToolResult BlockKind = "tool_result"
	// KindOpaque is provider-specific content the agent does not interpret but must
	// preserve and replay verbatim, such as a model's reasoning blocks that the
	// provider requires echoed back unchanged on the next turn. An adapter emits it
	// when decoding a response and splices its Raw bytes back when encoding a
	// request; the conversation loop carries it through untouched.
	KindOpaque BlockKind = "opaque"
	// KindImage is bitmap content in a user message: a pasted or attached image,
	// carried as raw bytes and a media type. A backend that supports vision encodes
	// it to its wire format; one that does not refuses the request rather than
	// dropping the image, so a picture never silently goes missing from a turn.
	KindImage BlockKind = "image"
)

type CacheHint

type CacheHint struct {
	// Prefix marks the system prompt and tool schemas as one cacheable boundary.
	// These are the largest reliably-stable region of a tool-using request.
	Prefix bool `json:"prefix,omitempty"`
	// StableMessages is the number of leading messages that are stable across turns.
	// A tool-using loop appends to its history and never edits an earlier turn, so
	// this is normally the full message count at call time: the previous turns are
	// frozen and worth caching, while only the newest content is unique to this call.
	// Zero leaves the message history uncached. A backend places at most one message
	// boundary, after message StableMessages-1, in addition to the prefix one.
	StableMessages int `json:"stableMessages,omitempty"`
	// Key is a stable identifier for the conversation whose prefix this request
	// shares, for example a run id. It is constant across the turns of one
	// conversation and distinct between conversations. A backend that routes by cache
	// affinity uses it to send same-prefix requests to the same place, raising the
	// hit rate; a backend without that mechanism ignores it. It is a routing hint,
	// not a secret, and never changes the request's content. Empty opts out.
	Key string `json:"key,omitempty"`
}

CacheHint tells a backend which parts of a request are stable across turns, so a provider that supports prompt caching can reuse the work of processing them instead of re-reading the whole prompt on every call. It is advisory and fully provider-neutral: the caller declares stability, which only it can know from how it assembles the conversation, and each adapter realizes the hint in its own terms. A provider with explicit cache markers places one at each declared boundary; a provider with automatic prefix caching, or one with no caching at all, ignores the hint and loses nothing, because the caller has kept the prefix byte-identical regardless. The hint can only ever save cost: it never changes what the model is asked or what it returns.

The boundaries run from most to least stable. The static prefix (the system prompt and tool schemas, identical every turn) is the cheapest large win; marking a count of leading messages adds the append-only conversation prefix on top of it. The zero value caches nothing, so a caller that does not opt in is unaffected.

type Image

type Image struct {
	MediaType string `json:"mediaType"`
	Data      []byte `json:"data"`
}

Image is bitmap content in a message: the raw encoded bytes and the IANA media type that types them (for example "image/png"). The bytes are carried inline, so an image travels with the turn it belongs to and survives a crash-resume in the checkpoint like the rest of the conversation. A vision backend encodes the bytes to its wire format; the media type tells it how.

type Message

type Message struct {
	Role   Role    `json:"role"`
	Blocks []Block `json:"blocks"`
}

Message is one turn in the conversation: a role and its ordered content blocks.

func Text

func Text(role Role, text string) Message

Text builds a single-block text message in the given role.

func (Message) TextContent

func (m Message) TextContent() string

TextContent concatenates the text blocks of a message, the human-readable answer with any tool-call blocks dropped.

func (Message) ToolUses

func (m Message) ToolUses() []ToolUse

ToolUses returns the tool calls the assistant requested in this message, in order, so the loop can execute them without re-walking block kinds at call sites.

type Model

type Model interface {
	Generate(ctx context.Context, req Request) (Response, error)
}

Model is the provider port: turn a Request into one assistant Response. It is the entire surface a backend implements and the only language-model dependency the rest of the agent has. Implementations should be safe for concurrent use and should return fault-classified errors (transient for rate limits and 5xx-style failures so the caller retries; terminal for malformed requests).

type Request

type Request struct {
	System    string    `json:"system,omitempty"`
	Messages  []Message `json:"messages"`
	Tools     []Tool    `json:"tools,omitempty"`
	MaxTokens int       `json:"maxTokens,omitempty"`
	Cache     CacheHint `json:"cache,omitempty"`
	// Sampling pins the decoding parameters for a reproducible run. Nil takes the server's
	// defaults (free-running); setting it sends a fixed seed and sampler so a local
	// generation can be recorded and replayed.
	Sampling *Sampling `json:"sampling,omitempty"`
}

Request is one call to a model: standing instructions, the conversation so far, the tools on offer, a ceiling on output length, and an optional hint about which of its parts are stable enough to cache. Other provider-specific knobs (thinking depth, sampling) are intentionally absent from the port; a backend applies its own sensible defaults, and a richer typed surface can be added behind the same interface if a real need appears.

type Response

type Response struct {
	Message    Message    `json:"message"`
	StopReason StopReason `json:"stopReason"`
	Usage      Usage      `json:"usage"`
}

Response is one model turn: the assistant message it produced, why it stopped, and what it cost.

type Role

type Role string

Role identifies who produced a message.

const (
	// RoleSystem carries standing instructions that frame the whole conversation.
	RoleSystem Role = "system"
	// RoleUser is input to the model: the task, and the results of tools it called.
	RoleUser Role = "user"
	// RoleAssistant is the model's own output: text and/or tool calls.
	RoleAssistant Role = "assistant"
)

type Sampling

type Sampling struct {
	// Seed fixes the generation's random seed. With a positive temperature, reproducibility
	// depends on the runtime honoring the seed; a recorded run carries it regardless so a
	// later replay can use the same value.
	Seed int64
	// Temperature scales randomness. Zero is greedy decoding, which yields the same output
	// every time on the same weights and is the strongest reproducible mode; higher is more
	// random. A negative or non-finite value normalizes to zero.
	Temperature float64
	// TopP is the nucleus-sampling cutoff in (0,1]. Zero leaves the server default in place; a
	// value outside [0,1] normalizes into it.
	TopP float64
}

Sampling pins the decoding parameters of a generation so a local run can be reproduced. A hosted API cannot promise this, but a model whose weights you hold can: with a fixed seed and greedy or seeded decoding, the same request yields the same output. A Request leaves Sampling nil to take the server's defaults (free-running); setting it opts into pinned decoding, which is what makes a local run a recordable, repeatable artifact.

func (Sampling) Deterministic

func (s Sampling) Deterministic() bool

Deterministic reports whether this sampling is in the guaranteed-reproducible regime. Greedy decoding (temperature zero) yields identical output on identical weights every time. With a positive temperature, reproducibility additionally needs the runtime to honor the seed, which is not guaranteed across runtimes, so that case is not reported as deterministic here.

func (Sampling) Normalized

func (s Sampling) Normalized() Sampling

Normalized returns a copy with every field forced into its valid range: temperature floored at zero, top-p clamped to [0,1], and any non-finite value dropped, so a malformed sampling can never reach a backend as a garbage value.

type StopReason

type StopReason string

StopReason is why the model ended its turn. It drives the conversation loop: ToolUse means run the requested tools and continue; EndTurn means the model is done and the turn is final.

const (
	// StopEndTurn means the model finished its turn with a final answer.
	StopEndTurn StopReason = "end_turn"
	// StopToolUse means the model wants to call one or more tools before continuing.
	StopToolUse StopReason = "tool_use"
	// StopMaxTokens means output hit the length ceiling and was cut off.
	StopMaxTokens StopReason = "max_tokens"
)

type Tool

type Tool struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	InputSchema json.RawMessage `json:"inputSchema"`
}

Tool describes a capability the model may call: a name, a description the model uses to decide when to call it, and a JSON Schema for its arguments. It is the declaration only; execution is the caller's (see the mission package).

type ToolResult

type ToolResult struct {
	ToolUseID string `json:"toolUseID"`
	Content   string `json:"content"`
	IsError   bool   `json:"isError,omitempty"`
}

ToolResult carries a tool's output back to the model. ToolUseID matches the ToolUse it answers; IsError marks a failed call so the model can adapt rather than mistake an error string for a successful result.

type ToolUse

type ToolUse struct {
	ID    string          `json:"id"`
	Name  string          `json:"name"`
	Input json.RawMessage `json:"input"`
}

ToolUse is a model's request to invoke a tool. ID correlates this call with the ToolResult that answers it, so parallel calls in one turn stay matched.

type Usage

type Usage struct {
	InputTokens     int `json:"inputTokens"`
	OutputTokens    int `json:"outputTokens"`
	CacheReadTokens int `json:"cacheReadTokens,omitempty"`
	// CacheWriteTokens is tokens written to the cache this call (a one-time cost a
	// later turn recovers). Providers with automatic prefix caching do not report a
	// separate write, so this stays zero for them.
	CacheWriteTokens int `json:"cacheWriteTokens,omitempty"`
}

Usage reports the token cost of a call, so a caller can meter spend and enforce budgets. Zero is a valid "unknown/unreported" value for backends that do not surface counts.

The cache fields are defined so they mean the same thing across providers even though providers report them differently. InputTokens is always the total input processed this call, INCLUDING any part served from cache. CacheReadTokens is the subset of InputTokens that the provider served from a prompt cache (billed at a large discount, so it is the win to measure: cache-hit-rate is CacheReadTokens over InputTokens). CacheWriteTokens is input written into the cache this call so a later call can reuse it; it is additional, not part of InputTokens, and on most providers is billed at a small premium. An adapter normalizes its provider's native counters into this shape, so a caller computes hit-rate and cost the same way regardless of which model ran.