llm

Documentation

Overview

Package llm is the metered model-execution gateway: the ONE path every model call goes through. Calling a model directly via provider.Complete/Stream is banned (a guard test enforces it) — instead callers hold a *Runner and pass a Purpose, so token/cost accounting (and, later, latency, route metadata, cost guardrails, and learning metrics) is automatic and impossible to bypass.

Architecture: provider (wire) → Runner (meters every call) → Ledger (one shared record) → /cost, /route, guardrails, learning. One Runner per top-level session; sub-agents share it, so their spend lands in the same Ledger with no manual rollup.

Index

• type BackendStat
• type Ledger
  • func (l *Ledger) ByBackend() map[string]BackendStat
  • func (l *Ledger) ByPurpose() map[Purpose]Stat
  • func (l *Ledger) Total() Stat
• type ModelRunner
• type Purpose
• type Runner
  • func NewRunner(prov provider.ModelProvider) *Runner
  • func (r *Runner) Complete(ctx context.Context, p Purpose, req wire.Request) (wire.Response, error)
  • func (r *Runner) Fork() *Runner
  • func (r *Runner) Ledger() *Ledger
  • func (r *Runner) Meter(p Purpose, reqModel string, resp wire.Response, latency time.Duration)
  • func (r *Runner) Provider() provider.ModelProvider
  • func (r *Runner) Stream(ctx context.Context, p Purpose, req wire.Request, h wire.StreamHandler) (wire.Response, error)
• type Stat

Types

type BackendStat

type BackendStat struct {
	Calls     int
	In, Out   int
	USD       float64        // actual at the served model's rate card
	LatencyMS int64          // summed wall-clock; divide by Calls for the mean
	Reasons   map[string]int // fallback reasons (cheap_lane_error/cheap_lane_overflow/…)
}

BackendStat aggregates per serving backend (fireworks | anthropic | openai | …) — where the tokens actually ran and what they cost at the served model's rate card. Every lane is token-billed (Fireworks + frontier APIs; the self-hosted/GPU-hours era is gone).

type Ledger

type Ledger struct {
	// contains filtered or unexported fields
}

Ledger is the single, concurrency-safe record of all model usage for a run. Shared across a session and its sub-agents.

func (*Ledger) ByBackend

func (l *Ledger) ByBackend() map[string]BackendStat

ByBackend returns a snapshot of per-backend usage — who actually served the session's calls, at what latency, and what the per-token bill would have been (for /cost backends and the GPU-economics decision).

func (*Ledger) ByPurpose

func (l *Ledger) ByPurpose() map[Purpose]Stat

ByPurpose returns a snapshot of per-purpose usage (for /cost --by-purpose, /route, and learning).

func (*Ledger) Total

func (l *Ledger) Total() Stat

Total returns the aggregate usage + cost across every metered call.

type ModelRunner

type ModelRunner interface {
	Complete(ctx context.Context, p Purpose, req wire.Request) (wire.Response, error)
	Stream(ctx context.Context, p Purpose, req wire.Request, h wire.StreamHandler) (wire.Response, error)
	Ledger() *Ledger
}

ModelRunner is the ONLY interface a model consumer should depend on. Every call carries a Purpose and is metered. (Consumers take this, never provider.ModelProvider.)

type Purpose

type Purpose string

Purpose labels WHY a model call was made, so cost/latency can be attributed per kind of work (and a future router can learn which purposes Haiku handles well).

const (
	MainLoop Purpose = "main_loop" // the executive turn loop
	Reflect  Purpose = "reflect"   // plan-mode reflection gate
	Synth    Purpose = "synth"     // plan synthesis
	Explore  Purpose = "explore"   // read-only scout sub-agent
	Overview Purpose = "overview"  // current-state synthesis
	Classify Purpose = "classify"  // cheap routing/safety classifier
	Predict  Purpose = "predict"   // next-step prediction
	Learn    Purpose = "learn"     // claim/knowledge extraction
	Route    Purpose = "route"     // turn router (future)
	Compact  Purpose = "compact"   // context compaction (future)
	Other    Purpose = "other"     // unlabelled — still counted
)

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner is the metered gateway over a provider.ModelProvider.

func NewRunner

func NewRunner(prov provider.ModelProvider) *Runner

NewRunner wraps a provider with a fresh ledger. Construct ONE at the top level (the cmd boundary) and thread it everywhere — sub-agents share it.

func (*Runner) Complete

func (r *Runner) Complete(ctx context.Context, p Purpose, req wire.Request) (wire.Response, error)

Complete runs a non-streamed call and meters it (usage, cost, latency, backend).

func (*Runner) Fork

func (r *Runner) Fork() *Runner

Fork returns a NEW Runner that shares this one's Ledger (and provider connection) but is otherwise its own object. Sub-agents Fork the parent's runner instead of sharing the pointer — so the only thing shared across the main loop and its scouts is the passive central Ledger, never an executor with per-context state.

func (*Runner) Ledger

func (r *Runner) Ledger() *Ledger

Ledger exposes the shared usage record.

func (*Runner) Meter

func (r *Runner) Meter(p Purpose, reqModel string, resp wire.Response, latency time.Duration)

Meter records usage for a call made OUTSIDE Complete/Stream — a side-channel like the advisor or the web tools — so its spend lands in the same shared Ledger. reqModel is the counterfactual basis (usually the same as the served model for a side-channel).

func (*Runner) Provider

func (r *Runner) Provider() provider.ModelProvider

Provider returns the wrapped provider — ONLY for capability assertions that aren't model calls (e.g. provider.WebSearcher / WebFetcher detection, doctor checks). Never use it to call Complete/Stream directly; that bypasses metering.

func (*Runner) Stream

func (r *Runner) Stream(ctx context.Context, p Purpose, req wire.Request, h wire.StreamHandler) (wire.Response, error)

Stream runs a streamed call (live deltas via h) and meters the final usage. Falls back to Complete if the underlying provider can't stream.

type Stat

type Stat struct {
	Calls                          int
	In, Out, CacheRead, CacheWrite int
	USD                            float64
}

Stat is an aggregate of usage + estimated cost (total or per-purpose).