Documentation
¶
Overview ¶
Package llm is a minimal streaming client for OpenAI-compatible chat completions APIs.
Index ¶
Constants ¶
const DefaultMaxAttempts = 8
DefaultMaxAttempts is the built-in retry budget for transient request failures (one initial try plus retries). Client.MaxRetries overrides it; exported so the UI can show "attempt N/M".
Variables ¶
This section is empty.
Functions ¶
func IsContextLimit ¶
IsContextLimit reports whether err is a context-length-exceeded style error: an HTTP 4xx whose body names context length, or the older "context window"-free provider error code. It is the signal to auto-compact.
func SessionCost ¶
SessionCost returns the USD spend for cumulative usage u at per-token rates. Cached prompt tokens are billed at the cache-read rate when advertised, else at the full input rate (pi models.ts calculateCost has the same shape, plus a cache-write term OpenAI-compatible usage lacks).
Types ¶
type Client ¶
type Client struct {
BaseURL string
APIKey string
HTTP *http.Client
// MaxRetries caps retries of transient request failures. 0 uses
// DefaultMaxAttempts; 1 disables retries (a single attempt).
MaxRetries int
// OnRetry, when set, is invoked before each retry of a transient request
// failure. Optional — nil means silent retries.
OnRetry func(RetryEvent)
}
Client talks to one provider endpoint.
func (*Client) Complete ¶
Complete sends a non-streaming chat request and returns the assistant text content plus the reported usage. It's used internally by compaction's summary call, where streaming would just add UI noise for a one-shot synthesis.
func (*Client) Stream ¶
func (c *Client) Stream(ctx context.Context, req Request, onText, onThink func(string), onToolCall func(id, name, args string)) (Message, Usage, error)
Stream sends the request and invokes onText for each content delta, onThink for each reasoning_content delta (both may be nil), and onToolCall for each tool-call state change as it streams (id/name/args snapshots; may be partial mid-stream). It returns the final assistant message (with any accumulated tool calls) plus the usage the provider reports on the terminal chunk (stream_options:include_usage).
Transient failures (transport errors, 429, 5xx) are retried with backoff — but only until the first visible delta has been handed to onText/onThink. After that point a retry would replay text the caller already rendered, so the error is surfaced instead. A retry regenerates the whole assistant message server-side; nothing in the request messages is mutated by a failed attempt, so retrying is idempotent.
type ContentPart ¶
type ContentPart struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
ImageURL *struct {
URL string `json:"url"`
} `json:"image_url,omitempty"`
}
ContentPart is one element of a multimodal user message: either text or an image (as a data-URL). Kimi K3 and OpenAI vision models require `content` as an array of these parts rather than a plain string when images are attached. The wire shape is {"type":"text","text":...} and {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}.
func ImagePart ¶
func ImagePart(ext string, data []byte) ContentPart
ImagePart builds an image ContentPart from raw bytes and a format extension.
type HTTPError ¶
HTTPError is returned when the API responds with a non-2xx status. Body is the ( capped ) response payload so callers can match against provider- specific reason strings; Error() keeps the "<status>: <body>" shape the existing tests assert ( e.g. "... 401 ..." ).
type Message ¶
type Message struct {
Role string `json:"role"`
Content string `json:"content"`
Parts []ContentPart `json:"-"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
ToolCallID string `json:"tool_call_id,omitempty"`
// Name is the function name on role "tool" messages. OpenAI ignores it,
// but Moonshot/Kimi requires it ("tool messages need a resolvable tool
// name") — without it every tool-using turn 400s.
Name string `json:"name,omitempty"`
// Authored marks a user message the human actually typed and submitted, as
// opposed to one whip injected on their behalf (steered background-task
// results, goal-check continuations). Internal only — never sent to the
// provider. Used so input-history recall cycles only real submissions.
Authored bool `json:"authored,omitempty"`
// SentAt is when the human submitted the message (local time). Internal
// only — never sent to the provider; used by the rewind picker's
// per-message timestamp. A pointer so omitempty drops it for injected and
// pre-field messages (a zero time.Time struct is never omitted).
SentAt *time.Time `json:"sent_at,omitempty"`
// Usage is the token accounting for the assistant response that produced
// this message. Internal only — never sent to the provider; powers
// per-turn cost display and survives session resume (the in-memory
// session totals do not).
Usage *Usage `json:"usage,omitempty"`
// Model records which model produced an assistant message ("id @
// provider"), so a /model switch mid-session doesn't rewrite history
// silently. Internal only — never sent to the provider.
Model string `json:"model,omitempty"`
// RewoundFrom notes that this message replaced an earlier clipped one
// (rewind + resubmit). Internal only — never sent to the provider.
RewoundFrom string `json:"rewound_from,omitempty"`
}
Message is one chat message. Content is a string; ToolCalls set on assistant messages, ToolCallID on role "tool" results. A user message may also carry image Parts (multimodal/vision) — when Parts is non-empty it is sent as the content array and Content is mirrored as a text part so both stay in sync.
func (Message) MarshalJSON ¶
MarshalJSON sends Content as a plain string for text-only messages and as a content-parts array (text + images) for multimodal ones.
func (Message) TextContent ¶
TextContent returns the message's text, whether it was set directly (Content) or carried in a Parts array (multimodal messages mirror their text into both).
func (*Message) UnmarshalJSON ¶
UnmarshalJSON accepts both the plain-string and content-parts wire forms.
type ModelInfo ¶
type ModelInfo struct {
ID string `json:"id"`
ContextLength int `json:"context_length,omitempty"`
MaxCompletionTokens int `json:"max_completion_tokens,omitempty"`
ReasoningEfforts []string `json:"reasoning_efforts,omitempty"`
Pricing *Pricing `json:"pricing,omitempty"`
// InputModalities lists the input types the model accepts (OpenRouter
// shape: ["text","image"]). Nil when the provider doesn't advertise it.
InputModalities []string `json:"input_modalities,omitempty"`
}
ModelInfo is one entry from the provider's GET /models list. Fields beyond the OpenAI spec (context_length, reasoning_efforts, pricing) are omitted by APIs that don't supply them.
func (ModelInfo) SupportsVision ¶
SupportsVision reports whether the model advertises image input.
type Pricing ¶
type Pricing struct {
Prompt string `json:"prompt"`
Completion string `json:"completion"`
InputCacheRead string `json:"input_cache_read,omitempty"`
}
Pricing is the provider's per-token USD rates as decimal strings (inference.net / OpenRouter shape). Nil Pricing on ModelInfo means the provider doesn't advertise prices.
type Request ¶
type Request struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
Tools []Tool `json:"tools,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
// Temperature and TopP are optional per-model sampling knobs. Pointers so
// 0.0 (a legitimate value) is distinguishable from unset; nil omits the
// field from the request, preserving provider defaults.
Temperature *float64 `json:"temperature,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
Stream bool `json:"stream"`
StreamOptions *struct {
IncludeUsage bool `json:"include_usage"`
} `json:"stream_options,omitempty"`
}
Request is a chat completions request.
type RetryEvent ¶
type RetryEvent struct {
Attempt int // the attempt that just failed (1-based)
Max int // total attempts the client will make (initial + retries)
Delay time.Duration // how long the client will sleep before retrying
Err error // the transient error that caused the retry
}
RetryEvent describes one failed attempt that is about to be retried. It is passed to the Client.OnRetry hook so the UI can show "retrying in Ns" instead of looking hung.
type Tool ¶
type Tool struct {
Type string `json:"type"`
Function struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters json.RawMessage `json:"parameters"`
} `json:"function"`
}
Tool is a tool definition advertised to the model.
type ToolCall ¶
type ToolCall struct {
ID string `json:"id"`
Type string `json:"type"`
Function struct {
Name string `json:"name"`
Arguments string `json:"arguments"`
} `json:"function"`
DurationMs int64 `json:"duration_ms,omitempty"`
ExitCode int `json:"exit_code,omitempty"`
}
ToolCall is a model-requested tool invocation. DurationMs and ExitCode are whip-internal execution bookkeeping (never sent to the provider): how long the tool ran and how it finished, for a future /tools perf view.
type Usage ¶
type Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
// PromptTokensDetails nests the cache hit count (OpenAI-compatible).
PromptTokensDetails *struct {
CachedTokens int `json:"cached_tokens"`
} `json:"prompt_tokens_details,omitempty"`
}
Usage is the token accounting the provider reports for one request (prompt = input, completion = output). CachedTokens counts the slice of the prompt served from the provider's prompt cache. Providers that omit usage leave all fields zero — the session totals just skip those calls.
Source Files