llm

Documentation

Overview

Package llm provides an OpenAI-compatible HTTP client using only stdlib.

Package llm provides an OpenAI-compatible HTTP client using only stdlib.

Index

• func ApplyCacheMarkers(messages []Message) ([]Message, []SystemBlock)
• func DiscoverModelContext(baseURL, apiKey, model string) int
• func ResetModelCache()
• type CacheControl
• type CallParams
• type CallResult
• type Client
  • func New(baseURL, apiKey, model, thinking string, thinkingBudget int, ...) *Client
  • func NewWithMaxTokens(baseURL, apiKey, model, thinking string, thinkingBudget int, maxTokens int, ...) *Client
  • func (c *Client) Call(ctx context.Context, messages []Message, systemBlocks []SystemBlock, ...) (*CallResult, error)
  • func (c *Client) CallStream(ctx context.Context, messages []Message, systemBlocks []SystemBlock, ...) (*CallResult, error)
  • func (c *Client) IsAnthropic() bool
  • func (c *Client) RequestTimeout() time.Duration
  • func (c *Client) SimpleCall(ctx context.Context, systemPrompt, userPrompt string) (string, error)
• type Delta
• type DeltaKind
• type FunctionDef
• type Message
• type StreamAbortedError
  • func (e *StreamAbortedError) Error() string
  • func (e *StreamAbortedError) Unwrap() error
• type SystemBlock
• type ThinkingConfig
• type ToolCall
• type ToolDef

Functions

func ApplyCacheMarkers

func ApplyCacheMarkers(messages []Message) ([]Message, []SystemBlock)

ApplyCacheMarkers annotates messages with Anthropic-style cache_control markers to enable prompt caching. It:

1. Marks the first system message (if present) with cache_control: ephemeral
2. Marks the first user message with cache_control: ephemeral

Returns the updated messages and a System field (populated if the system message was moved out of the messages array for Anthropic compatibility). Callers must only use this when the target provider is Anthropic (see Client.IsAnthropic) — OpenAI rejects the resulting request shape.

func DiscoverModelContext

func DiscoverModelContext(baseURL, apiKey, model string) int

DiscoverModelContext queries the /models endpoint of the configured base URL to discover the context window for the given model. Returns 0 if the endpoint doesn't support model attribute discovery or the model isn't found.

Results are cached per (baseURL, apiKey) so multiple agents using the same provider share a single API call. The full model list is cached so different model lookups from the same endpoint don't re-query.

Call this at startup before creating the engine. The HTTP call uses a 5s timeout and never blocks startup for more than that.

func ResetModelCache

func ResetModelCache()

ResetModelCache clears the model discovery cache. Used in tests.

Types

type CacheControl

type CacheControl struct {
	Type string `json:"type"` // "ephemeral"
}

CacheControl marks a message or system block as cacheable by Anthropic. Only send it to Anthropic endpoints: OpenAI rejects Anthropic-style request shapes with a 400 (e.g. the top-level "system" field), so the loop only applies cache markers when Client.IsAnthropic reports true.

type CallParams

type CallParams struct {
	Model           string          `json:"model"`
	Messages        []Message       `json:"messages"`
	System          []SystemBlock   `json:"system,omitempty"` // Anthropic-style system blocks
	Tools           []ToolDef       `json:"tools,omitempty"`
	Stream          bool            `json:"stream"`
	StreamOptions   *streamOptions  `json:"stream_options,omitempty"` // streaming only: request usage in-stream (OpenAI dialect)
	MaxTokens       int             `json:"max_tokens,omitempty"`     // max output tokens (0 = omit/provider default)
	Temperature     *float64        `json:"temperature,omitempty"`    // 0–2, nil = provider default
	Thinking        *ThinkingConfig `json:"thinking,omitempty"`
	ReasoningEffort string          `json:"reasoning_effort,omitempty"`
}

CallParams is the request body for /chat/completions.

type CallResult

type CallResult struct {
	Content          string     // assistant text
	ReasoningContent string     // DeepSeek reasoning/thinking tokens
	ToolCalls        []ToolCall // tool calls requested by the model
	InputTokens      int        // prompt_tokens from API usage (0 = not reported)
	OutputTokens     int        // completion_tokens from API usage (0 = not reported)

	// Cache metrics. Only populated when the provider returns them.
	// Anthropic: cache_creation_input_tokens, cache_read_input_tokens
	// OpenAI: prompt_tokens_details.cached_tokens
	// DeepSeek: prompt_cache_hit_tokens (read), prompt_cache_miss_tokens (write)
	CacheCreationTokens int // Anthropic — tokens written to cache
	CacheReadTokens     int // Anthropic — tokens read from cache hit
	CachedTokens        int // OpenAI — cached tokens in prompt
	// CacheReported is true when the provider returned any cache metrics at
	// all; false means "no data", which is different from "0 tokens cached".
	CacheReported bool
}

CallResult is the parsed response from /chat/completions.

type Client

type Client struct {
	BaseURL        string
	APIKey         string
	Model          string
	Thinking       string  // "enabled", "disabled", "low", "medium", "high", or empty
	ThinkingBudget int     // max thinking tokens for Anthropic extended thinking (0 = use default 5000)
	MaxTokens      int     // max output tokens (0 = provider default)
	Temperature    float64 // 0 = use provider default, <0 = omit from request
	// contains filtered or unexported fields
}

Client sends chat completion requests to any OpenAI-compatible endpoint.

func New

func New(baseURL, apiKey, model, thinking string, thinkingBudget int, timeout time.Duration) *Client

New creates a Client with the given timeout. Pass 0 to use the default (120s). The timeout applies per HTTP request — the agent loop may have multiple requests; set a generous timeout for deep-reasoning models.

func NewWithMaxTokens

func NewWithMaxTokens(baseURL, apiKey, model, thinking string, thinkingBudget int, maxTokens int, timeout time.Duration) *Client

NewWithMaxTokens creates a Client with a specific max_tokens setting. maxTokens=0 means no limit (provider default).

func (*Client) Call

func (c *Client) Call(ctx context.Context, messages []Message, systemBlocks []SystemBlock, tools []ToolDef) (*CallResult, error)

Call sends a chat completion request and returns the result. systemBlocks is optional — pass nil for providers that don't support the separate System field (OpenAI, DeepSeek). When non-nil, the system prompt is sent in the "system" field instead of as a system message in the messages array (Anthropic format for prompt caching).

func (*Client) CallStream

func (c *Client) CallStream(ctx context.Context, messages []Message, systemBlocks []SystemBlock, tools []ToolDef, cb func(Delta) error) (*CallResult, error)

CallStream sends a chat completion request with stream:true and delivers fragments to cb as they arrive, returning the fully assembled result — identical to what Call returns for the same logical response. cb is invoked synchronously from the reader; it must be non-blocking (same contract as loop.SignalHandler). Returning a non-nil error from cb aborts the stream and yields a *StreamAbortedError. A nil cb is allowed (assemble only). See the package comment for the timeout and fallback contract.

func (*Client) IsAnthropic

func (c *Client) IsAnthropic() bool

IsAnthropic reports whether the client's base URL targets the Anthropic API. Anthropic-specific request features (the top-level "system" field, cache_control markers) must only be sent when this is true — other providers reject them (OpenAI answers 400 unknown_parameter).

func (*Client) RequestTimeout

func (c *Client) RequestTimeout() time.Duration

RequestTimeout reports the per-request HTTP timeout the client was configured with. Callers that derive their own context deadlines for background LLM calls (extended memory) use it so their deadline never cuts a call off before the HTTP client itself would give up.

func (*Client) SimpleCall

func (c *Client) SimpleCall(ctx context.Context, systemPrompt, userPrompt string) (string, error)

SimpleCall sends a single-turn chat completion request and returns the text response. No tools, no streaming, no thinking config. Used for lightweight LLM calls like skill risk assessment.

type Delta

type Delta struct {
	Kind DeltaKind
	Text string
}

Delta is one streamed fragment. Text is the concatenated fragment for this event, not the accumulated text.

type DeltaKind

type DeltaKind int

DeltaKind discriminates streamed fragments.

const (
	// DeltaReasoning is a reasoning/thinking fragment (reasoning_content),
	// emitted before content on thinking models.
	DeltaReasoning DeltaKind = iota
	// DeltaContent is an assistant text fragment.
	DeltaContent
	// DeltaToolArgs is a tool-call argument fragment (partial JSON). The
	// engine suppresses these by default; they exist for consumers that
	// render live tool invocations.
	DeltaToolArgs
)

type FunctionDef

type FunctionDef struct {
	Name        string `json:"name"`
	Description string `json:"description"`
	Parameters  any    `json:"parameters"`
}

FunctionDef defines a single tool's function signature.

type Message

type Message struct {
	Role             string        `json:"role"`           // "system", "user", "assistant", "tool"
	Content          string        `json:"content"`        // text content
	Name             string        `json:"name,omitempty"` // tool name (for tool role)
	ToolCallID       string        `json:"tool_call_id,omitempty"`
	ToolCalls        []ToolCall    `json:"tool_calls,omitempty"`        // required for assistant role with tool calls
	ReasoningContent string        `json:"reasoning_content,omitempty"` // DeepSeek reasoning tokens, must be echoed back
	CacheControl     *CacheControl `json:"cache_control,omitempty"`     // Anthropic prompt caching marker
}

Message represents a chat message.

type StreamAbortedError

type StreamAbortedError struct {
	Reason error
}

StreamAbortedError is returned by CallStream when the delta handler aborted generation. It wraps the handler's error and carries the partial result assembled so far via the CallStream return values.

func (*StreamAbortedError) Error

func (e *StreamAbortedError) Error() string

func (*StreamAbortedError) Unwrap

func (e *StreamAbortedError) Unwrap() error

type SystemBlock

type SystemBlock struct {
	Type         string        `json:"type"` // "text"
	Text         string        `json:"text"`
	CacheControl *CacheControl `json:"cache_control,omitempty"`
}

SystemBlock represents an Anthropic-style system prompt block with optional cache control. OpenAI-compatible endpoints that don't support this format silently ignore the field.

type ThinkingConfig

type ThinkingConfig struct {
	Type         string `json:"type"`                    // "enabled" or "disabled"
	BudgetTokens int    `json:"budget_tokens,omitempty"` // Anthropic: max thinking tokens
}

ThinkingConfig controls extended thinking for DeepSeek and Anthropic models. Anthropic requires budget_tokens when type is "enabled"; DeepSeek ignores it.

type ToolCall

type ToolCall struct {
	ID       string `json:"id"`
	Type     string `json:"type"` // always "function"
	Function struct {
		Name      string `json:"name"`
		Arguments string `json:"arguments"`
	} `json:"function"`
}

ToolCall represents a single tool invocation requested by the model. Matches the OpenAI API format exactly.

type ToolDef

type ToolDef struct {
	Type     string      `json:"type"`
	Function FunctionDef `json:"function"`
}

ToolDef is the JSON Schema definition of a tool.