llm

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package llm is the OpenAI-compatible streaming chat client every agent role's Turn loop uses to reach the LLM. It supports streaming + non-streaming completions, tool calling via ToolDef / ToolCall, and per-call options through ChatOpts (notably ToolChoice for forcing a typed terminal tool). StreamChunk carries one token at a time plus the final Usage report; StreamChunkToSSE formats those for SSE relay to UI consumers.

Index

Constants

View Source 
const CallPrefix = "@@call "

CallPrefix introduces a tool call. Everything after it, up to the end of the JSON value, is the call's arguments as json-loose-heredoc.

View Source 
const DefaultBudgetMessage = "Thinking budget reached. Stop reasoning and give your final answer now."

DefaultBudgetMessage is what to say at the cut when the caller has no opinion. Imperative and short: it is injected mid-thought, so it competes with whatever the model was in the middle of saying.

View Source 
const HeredocEnd = "@@end"

HeredocEnd terminates a call block, and doubles as the stop sequence.

A grammar does not force EOS when it completes: measured with `root ::= call`, the model produced a valid call and then repeated it to the token cap. Pass HeredocStop as ChatOpts.Stop and re-append HeredocEnd before parsing, since the provider strips the matched stop string.

View Source 
const HeredocOpen = "~~~"

HeredocOpen introduces a raw body: `key: <ext>` + this + newline.

View Source 
const HeredocSentinel = "~~~AGENTKIT_EOF_7F3A"

HeredocSentinel terminates a raw body. It is deliberately long and unpronounceable: a body line must equal it exactly to end the block, so the only failure mode is content that contains this exact line. Short sentinels like "EOF" or "END" appear in real files and shell scripts.

It must not begin with "<". `<|` is a single token in Qwen-family vocabularies, and a delimiter starting with "<" pulls generation toward it: asked to emit `<<END` the model produced `<|<END`. Measured alternatives — `~~~END` and `@@END` reproduced exactly, `<<END` and `:::END` did not.

View Source 
const MaxBatchedCalls = 4

MaxBatchedCalls caps how many tool calls one response may carry.

More than one is the point: the native format can emit parallel calls and the heredoc parser always could, so restricting the grammar to one silently gave that up. Independent work — read three files, list two directories — costs one round trip instead of three.

Capped rather than unbounded because unbounded repetition is precisely what made the model loop: once a call completes the grammar accepts, but another call is equally legal, and nothing forces EOS. Four covers the realistic fan-out; a fifth is another turn.

Variables

View Source 
var HeredocTypes = []string{
	"go", "java", "js", "json", "md", "py", "sh", "sql", "txt", "xml", "yaml",
}

HeredocTypes is the closed set of body type tags the grammar admits. Closed on purpose: an open [a-z]+ let the model place an argument VALUE in the tag slot. `json` is load-bearing (it makes the body a real value rather than a string); the rest are descriptive and only reach the tool as "<key>_type".

Functions

func CloseHeredoc added in v0.3.0 

func CloseHeredoc(raw string) string

CloseHeredoc re-appends the terminator a stop sequence consumed, so the text can be parsed. It is deliberately explicit rather than making the parser tolerate a missing terminator: an unterminated block is how TRUNCATED output looks, and the parser must keep refusing that.

func DecodeErr added in v0.3.0 

func DecodeErr(s string) error

DecodeErr reports why s is not JSON, or nil if it is.

func HeredocGrammar added in v0.3.0 

func HeredocGrammar(tools []ToolDef) string

HeredocGrammar constrains generation to `@@call <name>` plus one json-loose-heredoc object, for exactly these tools.

The grammar earns its keep by removing choices the model was measured to get wrong. It cannot emit an unlisted tool name. It cannot invent a body delimiter (asked to produce `<<END` it produced `<|<END`, because `<|` is one token). And the object is a real JSON object, so a value that is a number or a nested object is expressible without a type tag.

Pass it as ChatOpts.Grammar with NO tools set: llama.cpp refuses a grammar alongside `tools`, which is exactly why this format is parsed from content. Pair it with ChatOpts.Stop — a completed grammar does NOT force EOS, and without a stop the model re-emits the same call to the token cap.

func HeredocStop added in v0.3.0 

func HeredocStop() []string

HeredocStop is the stop sequence to pair with HeredocGrammar.

func HeredocSystemPrompt added in v0.3.0 

func HeredocSystemPrompt(tools []ToolDef) string

func IsTruncationErr added in v0.3.0 

func IsTruncationErr(err error) bool

IsTruncationErr distinguishes "the input stopped early" from "the input is wrong". A *json.SyntaxError means the decoder read a byte it could not accept; an EOF means it ran out while still expecting more.

func ParseLooseHeredocJSON added in v0.3.0 

func ParseLooseHeredocJSON(s string) (string, error)

ParseLooseHeredocJSON rewrites one json-loose-heredoc value to strict JSON.

func RenderToolCall added in v0.3.0 

func RenderToolCall(name, arguments string) string

RenderToolCall writes a stored tool call back in the format the model produces, for replaying history under the heredoc transport.

History must be shown in the SAME dialect the model writes, or it reads its own past in a language it never used. Rendering it back as native tool_calls would also re-enter the XML path this format exists to avoid, and would require a `tools` array that heredoc mode deliberately does not send.

Values are re-encoded as ordinary JSON rather than as bodies: a raw body is only needed while GENERATING, to dodge escaping the model is bad at. Expanding one here would risk a delimiter collision for no benefit.

func StreamChunkToSSE 

func StreamChunkToSSE(chunk StreamChunk) string

StreamChunkToSSE formats a StreamChunk as an SSE event string.

func TransientUpstream added in v0.3.0 

func TransientUpstream(err error) bool

TransientUpstream reports whether an error is the SERVER going away rather than anything wrong with the request.

The distinction is not cosmetic. A benchmark that counts a redeploy as a task failure reports the wrong number, and an agent that treats one as a bad request rewrites a prompt that was fine. Measured case: a run died at 21:42:53 with 

agent: chat: stream error: stream ID 49; CANCEL; received from peer

and the serving process's start time was 21:51:55 — a build-and-restart window that killed every in-flight stream at its front. Two runs of a five-run arm were lost that way and looked, in the results table, exactly like the model failing the task.

Deliberately narrow. It matches only faults that say the connection or the backend died: an HTTP/2 CANCEL or GOAWAY, a reset or refused connection, an EOF mid-stream, and the gateway statuses. A 400, a 500 from argument parsing, or a model that produced nonsense are all the caller's problem and must stay visible.

func TransientUpstreamReason added in v0.3.0 

func TransientUpstreamReason(err error) string

TransientUpstreamReason names WHICH transient fault fired, for a log or a benchmark's result row. Empty when the error is not transient.

It exists so a run can be marked infrastructure-invalid with the evidence attached, rather than a human reading stderr hours later and guessing — which is how the deploy above was originally misread as the agent failing.

Types

type Backpressure added in v0.4.0 

type Backpressure struct {
	// Reason is the proxy's own classification. corrallm: "rejected" (queue full),
	// "queue-timeout" (waited and never got a slot), "spill" (routed elsewhere),
	// "exhausted" (budget/quota gone).
	Reason string
	// RetryAfter is when the server says to come back. Ok=false from
	// backpressureFrom means it did not say.
	RetryAfter time.Duration
	// Capacity / InFlight / Waiting describe the queue: total slots, slots busy,
	// and requests already waiting. Zero = not reported.
	Capacity, InFlight, Waiting int
	// Message is the server's human-readable line, when it sent one.
	Message string
}

Backpressure is the detail a fair-share proxy attaches to a 429.

corrallm's proxy answers a saturated backend with a fully actionable 429 — Retry-After plus X-RateLimit-Capacity / -InFlight / -Waiting headers and a JSON body carrying the same numbers and a reason ("rejected", "queue-timeout", "spill", "exhausted"). Until now the client read the one field it needed to sleep and discarded the rest, so a user waiting out a queue was told "429" when the server had said "4 of 4 slots busy, 2 requests ahead of you, come back in 10s". That difference is the whole gap between a hang and a wait.

Every field is optional: a plain OpenAI-style 429 fills in nothing but RetryAfter, and a consumer must treat zeros as "not reported".

func (Backpressure) Queued added in v0.4.0 

func (b Backpressure) Queued() bool

Queued reports whether the server described a queue we are standing in, i.e. whether the capacity numbers are worth showing a user.

func (Backpressure) String added in v0.4.0 

func (b Backpressure) String() string

String renders the queue state as a phrase, empty when nothing was reported.

type ChatOpts 

type ChatOpts struct {
	ToolChoice     string
	TraceID        string
	Grammar        string
	ResponseFormat any

	// Temperature and Seed pin sampling for reproducibility. Both are pointers
	// so "unset" stays distinct from a deliberate 0 — temperature 0 is exactly
	// the value a caller most wants to send, and a plain float64 could never
	// express it without also forcing it on every caller that had no opinion.
	//
	// A measurement harness needs these: without them the server's own sampling
	// config decides, so a model launched with --temp 0.7 makes a pass/fail
	// probe a coin flip and single-shot runs disagree with themselves.
	// Providers that do not support seed ignore the field.
	Temperature *float64
	Seed        *int

	// Stop ends generation when any of these strings is produced. The provider
	// excludes the matched string from the returned text.
	//
	// Needed with Grammar: a completed grammar does NOT force EOS. Measured with
	// `root ::= call`, the model finished a valid call and then emitted the same
	// call again, repeatedly, to the token cap. Stopping on the block terminator
	// is what actually ends the turn.
	Stop []string

	// MaxTokens caps the completion. 0 = leave it to the server.
	//
	// This matters most with Grammar set. A grammar that permits repetition (a
	// heredoc body is `bodyline*`) never REQUIRES termination, so an unbounded
	// request can run to the context limit instead of finishing — observed as a
	// hang, not an error.
	MaxTokens int

	// Repetition controls. Pointers for the same reason as Temperature: 0 is a
	// meaningful value (it is "off"), and a plain float64 could not distinguish
	// "the caller wants it off" from "the caller had no opinion".
	//
	// These exist because a CUT is not a cure. RepetitionGuard stops a runaway
	// after the fact; it does nothing about the cause, and at temperature 0 the
	// cause is deterministic — measured on one survey page, the same generation
	// was cut three times in 34 seconds with byte-identical output, because a
	// greedy decoder in a repeating basin has no way out. Only sampling does.
	//
	// FrequencyPenalty and PresencePenalty are the OpenAI-standard pair
	// (penalize by count seen, and by seen-at-all). RepeatPenalty is
	// llama.cpp's own multiplicative penalty over a trailing window, ignored by
	// providers that do not implement it — 1.0 is off, ~1.1 is a light touch.
	FrequencyPenalty *float64
	PresencePenalty  *float64
	RepeatPenalty    *float64

	// Reasoning budget. See reasoning.go for the provider mapping — the two
	// dialects use different fields and NEITHER rejects the other's, so a
	// hand-written body cannot tell a working budget from a dropped one.
	//
	// Pointer for the same reason as the penalties: 0 is a meaningful value
	// (llama.cpp reads it as "end reasoning immediately") and -1 means
	// disabled, so a plain int could not express either without also forcing a
	// choice on callers that have none.
	ReasoningBudgetTokens *int

	// ReasoningBudgetMessage is injected at the cut. Only llama.cpp has this;
	// Anthropic ends thinking without saying so. Empty sends nothing.
	ReasoningBudgetMessage string

	// EnableThinking overrides the SERVER's default mode per request, via
	// chat_template_kwargs. Needed because the server flag is only a default:
	// Qwen3.8 ships thinking ON, a deployment may launch it with
	// --reasoning off, and a caller that wants thinking has to ask for it.
	//
	// Meaningless to a template that does not read the kwarg, and unused on the
	// Anthropic dialect, where a budget is itself the toggle.
	EnableThinking *bool
}

ChatOpts carries per-call switches.

ToolChoice: "", "auto", "required", or a JSON-encoded 

{"type":"function","function":{"name":"foo"}}.
OpenAI-compatible providers honor "required" or a specific
tool spec.

TraceID: the value sent in the X-Trace-Id header, for correlation 

only — server-side logs can attribute requests back to a thread.
Scheduling priority is keyed off the API key (Authorization
Bearer), not this header. Harness Sessions set TraceID = thread
id; empty is fine (the header just won't be sent).

Grammar: when non-empty, forwarded as the request body's "grammar" 

field — a GBNF grammar the server constrains token sampling to
(llama.cpp / corrallm). Raw passthrough; the server owns the
syntax. Use for hard structural guarantees the model cannot
violate (vs the client-side agent.Validator fix loop, which
corrects a bad reply after the fact).

ResponseFormat: when non-nil, forwarded as the request body's 

"response_format" — e.g. map[string]any{"type":"json_object"} or a
{"type":"json_schema","json_schema":{...}} object. Marshaled
as-is; the server decides support.

Nil opts behaves as the default.

type Client 

type Client struct {

	// RetryBudget caps the total wall-clock a single request spends retrying
	// 429/5xx before giving up (exponential backoff to retryMaxBackoff,
	// honoring retry_after, but bounded). 0 → defaultRetryBudget (5m). The
	// caller's ctx deadline still wins if shorter. Set it per Client for a
	// busy endpoint.
	//
	// NEGATIVE means unbounded: ride out backpressure until the ctx says stop.
	// Correct for a single-slot endpoint, where a wait is not a delay but
	// another attempt at the only slot, and a 5-minute default gives up in the
	// middle of an ordinary busy spell.
	//
	// Unbounded is safe because it only extends the 429 path. 5xx is ALSO capped
	// by an attempt count (retry5xxMaxAttempts), so a genuinely broken upstream
	// still stops; and a 500 reporting unparseable tool-call arguments is
	// returned immediately without retrying, since the model produced those bytes
	// deterministically from this context and every retry reproduces them.
	// Nothing can spin forever on an outcome that cannot change.
	RetryBudget time.Duration

	// Retry5xxAttempts caps how many times a 5xx is retried. 0 → default (5).
	//
	// The default is tuned for a chat turn, where five failures means the
	// upstream is genuinely broken and an operator should hear about it. It is
	// wrong for a long BATCH: transcribing a 33-page document is ~33 requests,
	// and a single upstream blip outlasting ~15s of backoff fails the whole
	// document. Measured on a real corpus, every document over 20 pages failed
	// this way and no document over 20 pages ever completed.
	//
	// Raising it is safe because RetryBudget still bounds the wall clock — the
	// attempt cap and the budget are two different guards, and for batch work the
	// budget is the one that should bind.
	Retry5xxAttempts int

	// Repetition configures degenerate-loop detection on streamed output. The
	// zero value is ON with defaults; set Repetition.Off to disable. See
	// RepetitionGuard for why this is not opt-in.
	Repetition RepetitionGuard

	// OnRetry, when set, is called once per retry decision (and once when the
	// loop recovers or gives up), so a caller can SHOW the wait instead of
	// sitting mute through it. See RetryEvent.
	//
	// Called from the goroutine issuing the request, synchronously, while the
	// caller is blocked in Chat/ChatStream/Embed — so it must not block for long
	// and must not re-enter the client.
	OnRetry func(RetryEvent)
	// contains filtered or unexported fields
}

Client sends requests to an OpenAI-compatible LLM endpoint.

func NewClient 

func NewClient(baseURL, apiKey, model string) *Client

func (*Client) AutoReasoningBudget added in v0.5.0 

func (c *Client) AutoReasoningBudget(ctx context.Context, system string, tools []ToolDef, reserve int) (int, bool)

AutoReasoningBudget sizes a thinking budget from what is actually left in the context window, and ok=false when it cannot be sized honestly.

reserve is what the caller wants kept back for the final answer BEYOND the share arithmetic — tool results still to come, a compaction margin, whatever the caller knows and this package does not.

ok=false is a fact, not an error, and the caller should send no budget at all rather than guess one: an unbudgeted request thinks as much as it wants, which is right when the window is unknown. A guessed budget can only truncate.

func (*Client) Chat 

func (c *Client) Chat(ctx context.Context, messages []Message, tools []ToolDef) (string, []ToolCall, error)

Chat sends a non-streaming chat completion request.

func (*Client) ChatStream 

func (c *Client) ChatStream(ctx context.Context, messages []Message, tools []ToolDef, opts *ChatOpts) (<-chan StreamChunk, error)

ChatStream sends a chat completion request with streaming enabled. It returns a channel that emits StreamChunks as they arrive.

func (*Client) ChatWithOpts added in v0.3.0 

func (c *Client) ChatWithOpts(ctx context.Context, messages []Message, tools []ToolDef, opts *ChatOpts) (string, []ToolCall, error)

ChatWithOpts is Chat with sampling and constrained-decoding options.

func (*Client) ContextWindow added in v0.4.0 

func (c *Client) ContextWindow(ctx context.Context) (int, bool)

ContextWindow reports the model's context size as the SERVER states it, and ok=false when it does not state one.

Preferred over DiscoverContext wherever it answers: the probe costs O(log N) round trips and returns a lower bound, while this is one request for the number itself. llama.cpp reports it at /props as default_generation_settings.n_ctx.

func (*Client) CountPrompt added in v0.5.0 

func (c *Client) CountPrompt(ctx context.Context, system string, tools []ToolDef) (int, bool)

CountPrompt returns the exact token count of a system prompt plus a set of tool definitions, as this endpoint's model would actually see them, and ok=false when the endpoint cannot count.

This exists because the two kinds of endpoint count different things and the difference is not a detail. llama.cpp and vLLM tokenize a STRING: hand them text, get its tokens. Anthropic prices a RENDERED PROMPT — and the moment any tool is declared it adds its own tool-use preamble, measured at ~497 tokens on 2026-08-10 against a preamble-free baseline of 9: 

no tools   9      1 tool   551      2 tools   596

So on Anthropic the parts of a prompt are NOT independently countable and summable: counting each MCP server's schemas alone charges that preamble once per server. A caller wanting per-part costs has to take DIFFERENCES between whole-prompt counts, which is what the constant envelope in probeMessages makes exact.

On a string tokenizer this falls back to counting the concatenation, so one caller works against both and only the residual differs.

func (*Client) CountTokens added in v0.4.0 

func (c *Client) CountTokens(ctx context.Context, text string) (int, bool)

CountTokens returns the exact number of tokens `text` becomes for this client's model, and ok=false when the endpoint offers no tokenizer.

ok=false is a fact about the endpoint, not an error: the caller's job is to fall back to estimating, not to fail. A transport failure against a tokenizer that DID exist is also reported as ok=false — a count that could not be obtained is not a count, and blocking a document on it would trade a wrong size for no progress at all.

func (*Client) DiscoverContext added in v0.3.0 

func (c *Client) DiscoverContext(ctx context.Context) (int, error)

DiscoverContext probes the configured model's usable context window: it sends prompts that grow exponentially until the server rejects one as too long, then binary-searches to ~256-token resolution. It returns an approximate maximum INPUT token count (~1 token per probe word) — a safe LOWER BOUND on the real window. Costs O(log N) round-trips; cache the result rather than calling it per request.

IMPORTANT: this only finds a boundary on servers that REJECT an over-long prompt (llama.cpp direct, OpenAI, vLLM, TGI). Some proxies accept and process arbitrarily large prompts without error (corrallm was observed accepting 60k tokens with HTTP 200) — there is no boundary to find, so the probe stops at contextProbeCeiling and returns it as a lower bound. That keeps the probe cheap and bounded rather than climbing until requests get enormous.

func (*Client) Embed added in v0.3.0 

func (c *Client) Embed(ctx context.Context, model string, input []string) ([][]float32, error)

Embed returns one embedding vector per input string from an OpenAI-compatible /v1/embeddings endpoint. model is passed explicitly because embeddings use a different model than chat (the Client's configured model is the chat model). Output order matches input order. Honors the same auth + retry policy as Chat.

func (*Client) HasTokenizer added in v0.4.0 

func (c *Client) HasTokenizer(ctx context.Context) bool

HasTokenizer reports whether this endpoint can count tokens exactly, probing once. Useful for deciding a strategy before there is text to count.

func (*Client) RetryPolicy added in v0.4.0 

func (c *Client) RetryPolicy() (initial, max, budget time.Duration)

RetryPolicy reports the backoff schedule this client actually uses: the first delay, the ceiling it climbs to, and the wall-clock budget for one call.

It exists for a caller that must add an OUTER retry — the client's loop covers everything up to the response headers, but a stream that dies MID-generation is not resumable here, so only the caller can decide to run the turn again. Such a caller should not invent a second schedule out of thin air: the numbers (and any operator override of RetryBudget) are policy, and one policy is better than two that disagree.

These are the UNJITTERED schedule. This client adds random slack to each of its own sleeps (retryJitterFraction) so concurrent callers don't retry in lockstep; an outer retrier that fans out across sessions wants the same property and should jitter what it builds from these numbers.

func (*Client) WithAutoReasoningBudget added in v0.5.0 

func (c *Client) WithAutoReasoningBudget(ctx context.Context, opts *ChatOpts, system string, tools []ToolDef, reserve int) *ChatOpts

WithAutoReasoningBudget fills in ReasoningBudgetTokens on opts when it is unset, leaving a caller's explicit budget alone.

Deliberately does NOT set EnableThinking. Whether to think is a decision about the task; how long to think is a decision about the window, and only the second one is arithmetic this package can do.

type ContentPart added in v0.3.0 

type ContentPart struct {
	Type     string    `json:"type"`
	Text     string    `json:"text,omitempty"`
	ImageURL *ImageURL `json:"image_url,omitempty"`
}

ContentPart is one element of a multimodal content array. Type is "text" or "image_url"; exactly one of Text / ImageURL is set to match.

func ImageData added in v0.3.0 

func ImageData(mime string, raw []byte) ContentPart

ImageData builds an image content part from raw bytes, encoding them as a base64 data: URI with the given MIME type (e.g. "image/png"). This is the OCR path: rasterize/extract a page image, hand the bytes straight to a vision model without writing a temp file or hosting a URL.

func ImagePart added in v0.3.0 

func ImagePart(url string) ContentPart

ImagePart builds an image content part from a URL or data: URI.

func TextPart added in v0.3.0 

func TextPart(text string) ContentPart

TextPart builds a text content part.

type EchoResult added in v0.3.0 

type EchoResult struct {
	Name    string
	Want    string
	Got     string
	Lossy   bool
	NoCall  bool   // the endpoint returned no tool call at all
	BadJSON bool   // arguments did not parse
	Err     string // transport failure
}

EchoResult is one round trip. Lossy is the interesting column: it means the endpoint returned a tool call whose argument is not what the model was asked to send, with no error anywhere.

type ImageURL added in v0.3.0 

type ImageURL struct {
	URL    string `json:"url"`
	Detail string `json:"detail,omitempty"`
}

ImageURL is an image reference: either an https URL or an inline data: URI ("data:image/png;base64,…"). Detail ("auto"|"low"|"high") is optional.

type InputTooLargeError added in v0.4.0 

type InputTooLargeError struct {
	Status int
	Body   string
}

InputTooLargeError reports that the endpoint refused the REQUEST for its size — llama.cpp's physical batch, or a context that cannot hold the prompt.

Typed rather than a bare fmt.Errorf because two callers have to act on it and neither can match a sentence reliably. A caller sending a document has to send less, and DiscoverContext has to read it as "this size does not fit", which is the whole answer the probe is looking for — string-matched, it read as a transport failure and aborted the probe on the exact endpoint that needed it.

The Error() text is unchanged: it is already recorded in job rows.

func (*InputTooLargeError) Error added in v0.4.0 

func (e *InputTooLargeError) Error() string

type Message 

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
	Name    string `json:"name,omitempty"`
	// Parts, when non-empty, replaces Content with a multimodal content array
	// (text + image parts). Send-only: responses are always plain text, so this
	// is never populated on a decoded reply. Tagged "-" — MarshalJSON emits it
	// under "content" itself.
	Parts []ContentPart `json:"-"`
	// ToolCalls carries an assistant message's requested tool calls, so a
	// reconstructed conversation replays a valid assistant(tool_calls) →
	// tool(tool_call_id) structure instead of orphan tool messages.
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
	// ToolCallID links a role="tool" result back to the assistant tool call
	// that produced it (OpenAI requires this correlation).
	ToolCallID string `json:"tool_call_id,omitempty"`
}

Message represents a chat message in the OpenAI-compatible format.

Content is the plain-text body — the overwhelmingly common case, and the wire shape is a bare string. For MULTIMODAL input (a vision model reading an image — e.g. OCR), set Parts instead: when Parts is non-empty it OWNS the "content" field, which marshals as OpenAI's array-of-parts shape ([{type:text,...},{type:image_url,...}]) and Content is ignored. The string path is left byte-for-byte unchanged so every existing caller is unaffected.

func (Message) MarshalJSON added in v0.3.0 

func (m Message) MarshalJSON() ([]byte, error)

MarshalJSON renders "content" as a bare string (Parts empty, the default) or as the multimodal array (Parts set). Everything else marshals normally.

type PromptTokensDetails added in v0.3.0 

type PromptTokensDetails struct {
	CachedTokens int `json:"cached_tokens"`
}

PromptTokensDetails is OpenAI's nested prompt-token breakdown.

type RepairOutcome added in v0.3.0 

type RepairOutcome int

RepairOutcome is what a repair attempt concluded. RepairTruncated is separate from RepairFailed because the two produce different advice to the model. 

const (
	RepairFailed RepairOutcome = iota
	RepairOK
	RepairTruncated
)

func RepairLooseJSON added in v0.3.0 

func RepairLooseJSON(s string) (string, []string, RepairOutcome)

repairLooseArgs turns near-JSON tool arguments into JSON, or reports that it could not. It returns the repaired text, the list of repairs applied (for the log), and whether anything usable came out.

WHY REPAIR AT ALL: rejecting a call costs a full regeneration of its arguments, and generated tokens are the most expensive channel there is — roughly an order of magnitude over cached prompt. A trailing comma is not worth re-writing a 4KB file for.

WHAT IT WILL NEVER DO: close an unterminated string, brace or bracket. That is the whole discipline here. Off-the-shelf JSON-repair routines auto-close, which on a TRUNCATED payload fabricates a call the model never finished writing and hands it to a dispatcher as though it were complete — a half- written file, a delete with half a filter. Truncation is diagnosed BEFORE repair is attempted (see checkArgs) and is never sent here. Repair is for syntax the model got wrong, never for output that stopped early.

Every repair is also validated afterwards: anything that does not come out as a strict JSON object is discarded and the call is refused as if no repair had been attempted. A botched repair therefore degrades to the existing refusal, never to a wrong dispatch. RepairLooseJSON is exported because both the tool-argument gate (agent) and the heredoc json body reader (llm) need it, and a second copy of "never repair a truncation" is the last invariant that should be duplicated.

type RepetitionGuard added in v0.3.0 

type RepetitionGuard struct {
	// Off disables detection entirely.
	Off bool

	// MinPeriod is the shortest repeating block considered a loop (default 24).
	// Below this, ordinary formatting repeats freely.
	MinPeriod int
	// MaxPeriod is the longest block scanned for (default 1024). Cost is linear
	// in this, so it bounds the check rather than the phenomenon.
	MaxPeriod int
	// MinReps is how many consecutive identical copies are required (default 3).
	MinReps int
	// MinSpan is the minimum total bytes the repetition must cover (default
	// 512). This is what stops three copies of a short header from tripping it.
	MinSpan int
	// CheckEvery is how many newly-arrived bytes trigger a scan (default 256).
	// The scan is not run per delta: a delta is a token, and scanning per token
	// would multiply the cost by ~4 for no earlier detection than this.
	CheckEvery int
}

RepetitionGuard configures degenerate-loop detection for a Client's streams. The zero value means "on, with defaults" — the failure it prevents is expensive and silent, so it is not opt-in. Set Off to disable it.

The thresholds exist to separate a LOOP from legitimate repetition. Real output repeats constantly: markdown table rules, blank lines, boilerplate headers, near-identical rows. What a loop adds is scale — the same bytes, exactly, for a long span with nothing else between. Requiring several copies AND a non-trivial period AND a large total span is what makes a false positive cost real work to construct, while the measured case (an 85-byte period, hundreds of copies) trips within the first kilobyte.

type RepetitionInfo added in v0.3.0 

type RepetitionInfo struct {
	// Where the loop was detected: "content", or "tool:<name>" for a call's
	// arguments.
	Where string
	// Period is the length in bytes of the repeating block; Reps is how many
	// consecutive copies of it were seen.
	Period int
	Reps   int
	// Trailing is the byte count of redundant copies — (Reps-1)*Period — i.e.
	// how much can be trimmed from the end of the accumulated text while leaving
	// one intact copy of the block.
	Trailing int
	// Sample is one copy of the repeating block, capped for logging.
	Sample string
}

RepetitionInfo describes the loop that was cut: enough to log it, to tell the model what it did, and to trim the redundant copies before they are persisted into the next context (which would otherwise TEACH the loop).

func (*RepetitionInfo) String added in v0.3.0 

func (r *RepetitionInfo) String() string

String renders the finding for a log line or a message to the model.

type RetryEvent added in v0.4.0 

type RetryEvent struct {
	Kind RetryKind
	// Attempt is the 1-based attempt this event describes: the one that just
	// failed, or — for RetryRecovered — the one that succeeded.
	Attempt int
	// Status is the HTTP status, 0 on the transport path (no response at all).
	Status int
	// Err is the transport error, nil on the status paths.
	Err error
	// Body is the first line of the server's own message, when it sent one. The
	// single most actionable field on a 5xx (see the "input too large" case in
	// postWithRetry) and empty far more often than not.
	Body string
	// Delay is how long the client will wait before the next attempt. 0 on
	// RetryRecovered / RetryGiveUp.
	Delay time.Duration
	// Elapsed is time spent in this call's retry loop so far.
	Elapsed time.Duration
	// Budget is the wall-clock ceiling for the whole loop (Client.RetryBudget).
	Budget time.Duration
	// Attempts5xx / Max5xx expose the SEPARATE attempt cap that bounds the 5xx
	// path, which is what actually ends most upstream outages — the budget rarely
	// gets a chance to. Both 0 outside the 5xx path.
	Attempts5xx, Max5xx int
	// BP carries the queue detail a fair-share proxy attached to a 429 — slots
	// busy, requests ahead, why. Non-nil only on the 429 path, and only worth
	// rendering when BP.Queued().
	BP *Backpressure
	// ServerAsked is true when Delay is what the SERVER instructed (Retry-After or
	// a backpressure body) rather than our own exponential schedule. Worth saying
	// out loud: it means the wait is a real estimate, not a guess.
	ServerAsked bool
	// Reason is a human-readable sentence: what failed and what happens next.
	Reason string
}

RetryEvent is one report from the client's backoff loop, delivered to Client.OnRetry.

It exists because the retry logic was already correct and completely INVISIBLE: every decision was a log.Printf, and a caller whose logs go anywhere but the user's screen (a TUI, a daemon, anything with a log file) shows a frozen cursor for up to the whole retry budget and then one error line. The wait is the part the user needs narrated — it is the only time the agent is doing nothing and the only time a human might reasonably intervene.

Every field is filled on a best-effort basis; a consumer should render Reason (always set) and treat the rest as detail.

func (RetryEvent) String added in v0.4.0 

func (e RetryEvent) String() string

String renders the event as one line, the same shape the client's own log lines use.

func (RetryEvent) Unbounded added in v0.4.0 

func (e RetryEvent) Unbounded() bool

Unbounded reports whether the retry budget is effectively "until the caller gives up" (Client.RetryBudget < 0), so a UI renders "no limit" instead of a nonsense century.

type RetryKind added in v0.4.0 

type RetryKind string

RetryKind names WHY the client is backing off, so a UI can say something more useful than "retrying". 

const (
	// RetryTransport — the request never got a response: the gateway is down,
	// restarting, or unreachable.
	RetryTransport RetryKind = "transport"
	// Retry429 — backpressure. The server will serve, just not yet.
	Retry429 RetryKind = "429"
	// Retry5xx — the upstream answered with a server error.
	Retry5xx RetryKind = "5xx"
	// RetryRecovered — a later attempt succeeded. Fires once, so a UI that put up
	// a "retrying" banner knows to take it down.
	RetryRecovered RetryKind = "recovered"
	// RetryGiveUp — the client stopped retrying (attempt cap, wall-clock budget,
	// or a fault that no amount of waiting changes). The call's error follows.
	RetryGiveUp RetryKind = "giveup"
)

type Session 

type Session struct {
	ID       string
	Messages []Message
	// contains filtered or unexported fields
}

Session manages the message history and tool configuration for one LLM conversation.

func NewSession 

func NewSession(id, system string) *Session

func (*Session) AddAssistantMessage 

func (s *Session) AddAssistantMessage(content string)

func (*Session) AddToolCalls 

func (s *Session) AddToolCalls(calls []ToolCall)

func (*Session) AddToolMessage 

func (s *Session) AddToolMessage(toolCallID, content string)

func (*Session) AddToolResult 

func (s *Session) AddToolResult(toolCallID, result string)

AddToolResult adds a tool call from the assistant followed by the tool's result message.

func (*Session) AddUserMessage 

func (s *Session) AddUserMessage(content string)

func (*Session) ChatStream 

func (s *Session) ChatStream(ctx context.Context, client *Client) (<-chan StreamChunk, error)

ChatStream sends messages + tools to the LLM and returns the stream.

func (*Session) GetMessages 

func (s *Session) GetMessages() []Message

func (*Session) GetTools 

func (s *Session) GetTools() []ToolDef

func (*Session) SetTools 

func (s *Session) SetTools(tools []ToolDef)

func (*Session) String 

func (s *Session) String() string

func (*Session) TrimMessages 

func (s *Session) TrimMessages(max int)

TrimMessages removes oldest messages to stay within context window.

type StopReason added in v0.3.0 

type StopReason string

StopReason names why a stream ended other than the model finishing. Empty on a normal completion, so a consumer that ignores it behaves as before. 

const StopReasonRepetition StopReason = "repetition"

StopReasonRepetition means the client cut the stream because generation had collapsed into a repeating block. The output up to the cut is still valid; what follows would have been more copies of the same bytes.

type StreamChunk 

type StreamChunk struct {
	Content  string
	ToolCall *ToolCall
	Done     bool
	Error    string
	Usage    *Usage

	// PartialToolCalls carries whatever tool-call arguments had accumulated
	// when an Error chunk was produced. Providers stream tool arguments
	// incrementally, so a call cut off mid-argument (a context window filling
	// mid-write) is already buffered client-side — the caller can salvage the
	// work instead of making the model regenerate it.
	//
	// These are INCOMPLETE and deliberately NOT delivered as ToolCall chunks:
	// dispatching a truncated call would run it with half its arguments.
	// Only ever set alongside Error.
	PartialToolCalls []ToolCall

	// StopReason names a non-model reason the stream ended — set on the Done
	// chunk. Empty on a normal completion, so consumers that ignore it are
	// unaffected. Today the only value is StopReasonRepetition.
	StopReason StopReason

	// Repetition describes the degenerate loop that was cut, when StopReason is
	// StopReasonRepetition. It carries what to log, what to tell the model, and
	// how many trailing bytes are redundant copies. Nil otherwise.
	Repetition *RepetitionInfo
}

StreamChunk is one token from the streaming response. The final chunk carries Usage (when the provider supports include_usage); all other fields will be zero on that chunk.

type ToolCall 

type ToolCall struct {
	ID       string `json:"id"`
	Type     string `json:"type"`
	Function struct {
		Name      string `json:"name"`
		Arguments string `json:"arguments"`
	} `json:"function"`
}

ToolCall represents a tool call request from the LLM.

Arguments is a STRING holding JSON, per the OpenAI spec — and per that same spec it is not guaranteed to be valid JSON, so every consumer must parse defensively (agent.Session refuses to dispatch a call it cannot parse). Marshalling keeps the string form, which is what a request must carry.

func ParseToolCalls added in v0.3.0 

func ParseToolCalls(content string) (calls []ToolCall, prose string, err error)

ParseToolCalls extracts tool calls from model content, plus any surrounding prose.

The shape is a prefix line naming the tool, then ONE json-loose-heredoc object: 

@@call write_file
{ path: "a.java", content: ~~~EOF
public class A {}
~~~EOF
}

Arguments arrive as strict JSON with real types, so a tool whose schema wants a number or an object gets one. That is the reason for parsing JSON rather than a line format: only STRING bodies are awkward to escape, so only string bodies get alternate syntax, and `{name: "prod", retries: 3}` stays two obvious pairs instead of two heredocs.

The object self-terminates, so no end marker is needed to parse. An UNCLOSED object, string or body is reported as truncation rather than closed silently.

func (*ToolCall) UnmarshalJSON added in v0.3.0 

func (tc *ToolCall) UnmarshalJSON(b []byte) error

UnmarshalJSON accepts `arguments` as either the spec's JSON string or a bare JSON object, which some providers send instead.

Without this, an object made the whole decode fail: the non-streaming path returned a decode error for the entire response, and the streaming path — which ignores unparseable events by design, since a provider may interleave shapes it does not recognise — dropped the chunk SILENTLY. Measured: zero tool calls, zero errors, no log. The turn saw a model that had said nothing, which is indistinguishable from a model that chose to say nothing.

type ToolDef 

type ToolDef struct {
	Type     string `json:"type"`
	Function struct {
		Name        string `json:"name"`
		Description string `json:"description"`
		Parameters  any    `json:"parameters"`
	} `json:"function"`
}

ToolDef describes a tool the LLM can call, matching the OpenAI tool format.

type ToolSurfaceReport added in v0.3.0 

type ToolSurfaceReport struct {
	Model string
	// Echo records one call-and-echo round trip per payload shape: the model is
	// asked to pass a known string through a tool argument, and the argument that
	// comes back is compared byte for byte.
	Echo []EchoResult
	// GrammarAccepted reports whether the endpoint accepts a GBNF grammar at all.
	GrammarAccepted bool
	// GrammarWithToolsAccepted reports whether a grammar may be combined with
	// tools. llama.cpp refuses ("Cannot specify grammar with tools"), which is
	// why a custom tool-call format parsed from content is the ONLY path where
	// generation can be constrained at the sampler.
	GrammarWithToolsAccepted bool
	GrammarError             string
}

ToolSurfaceReport is what an endpoint actually does with tool calls, measured rather than assumed. Two endpoints advertising the same OpenAI API differ on every line of this.

func ProbeToolSurface added in v0.3.0 

func ProbeToolSurface(ctx context.Context, c *Client) (ToolSurfaceReport, error)

ProbeToolSurface measures what an endpoint does with tool calls. It is cheap (one short generation per payload) and worth running once per model before trusting a tool loop against it: the failures it finds are all silent, so nothing else in the stack will report them.

Temperature and Seed are pinned so a re-run is comparable.

func (ToolSurfaceReport) OK added in v0.3.0 

func (r ToolSurfaceReport) OK() bool

OK reports whether every probed property held.

func (ToolSurfaceReport) String added in v0.3.0 

func (r ToolSurfaceReport) String() string

String renders the report as a short table for a log or a CLI.

type TruncatedToolCallError added in v0.3.0 

type TruncatedToolCallError struct {
	Status int
	Body   string
}

TruncatedToolCallError reports a response whose tool-call arguments could not be parsed because generation was CUT OFF — typically the context window filling mid-argument while the model writes a large file.

It is separated from ordinary 5xx precisely so a caller does not retry it: at temperature 0 the same context reproduces the same truncation byte for byte. The useful responses are to compact the context and re-issue, or to ask for a smaller edit — not to try again unchanged.

func (*TruncatedToolCallError) Error added in v0.3.0 

func (e *TruncatedToolCallError) Error() string

type Usage 

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`

	// Anthropic's shape: flat, top-level.
	CacheReadInputTokens     int `json:"cache_read_input_tokens,omitempty"`
	CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty"`

	// OpenAI's shape: NESTED. llama-server (and thus corrallm) speaks this
	// one, and only this one — it was previously unparsed, so every cached
	// token read as zero and prompt_tokens looked like real work forever.
	PromptTokensDetails *PromptTokensDetails `json:"prompt_tokens_details,omitempty"`

	// LatencyMS is wall-clock for this one round-trip, measured client-side.
	// Not reported by any provider — but "how long did that call take" is half
	// of what anyone asks of a trace, and without it the token counts alone
	// cannot distinguish a slow cold load from a large generation.
	LatencyMS int64 `json:"latency_ms,omitempty"`
}

Usage carries provider-reported token counts for one chat completion. Fields beyond prompt/completion/total are optional and only filled when the provider returns them (Anthropic's cache fields, OpenAI's reasoning tokens, etc).

PromptTokens counts the prompt the provider was SENT, not the prompt it actually had to process — a cached prefix is billed here in full even though it was never re-evaluated. Use CachedPromptTokens/NewPromptTokens for the real work; see the note on CachedPromptTokens for why that distinction is easy to get catastrophically wrong.

func (*Usage) CachedPromptTokens added in v0.3.0 

func (u *Usage) CachedPromptTokens() int

CachedPromptTokens returns prompt tokens served from cache, normalizing the two provider shapes.

This matters more than it looks. A cached prefix costs ~nothing to process (llama.cpp reuses the KV slot; measured ~1,860 tok/s prompt eval vs ~110 tok/s generation), but PromptTokens still reports it at full size on EVERY turn. Summing PromptTokens across a conversation therefore charges a stable prefix once per turn — which ranks the most-cached region of the prompt (the tool schemas, byte-identical every turn) as the dominant cost when it is very nearly free. That artifact drove a whole tool-surface redesign against a gap that, measured on generated tokens and wall clock, was ~12%.

func (*Usage) NewPromptTokens added in v0.3.0 

func (u *Usage) NewPromptTokens() int

NewPromptTokens returns the prompt tokens the provider actually had to evaluate this turn — the honest per-turn prompt cost.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.