llm

Documentation

Overview

Package llm is the metered model-execution gateway: the ONE path every model call goes through. Calling a model directly via provider.Complete/Stream is banned (a guard test enforces it) — instead callers hold a *Runner and pass a Purpose, so token/cost accounting (and, later, latency, route metadata, cost guardrails, and learning metrics) is automatic and impossible to bypass.

Architecture: provider (wire) → Runner (meters every call) → Ledger (one shared record) → /cost, /route, guardrails, learning. One Runner per top-level session; sub-agents share it, so their spend lands in the same Ledger with no manual rollup.

Index

• type BackendStat
• type Ledger
  • func (l *Ledger) ByBackend() map[string]BackendStat
  • func (l *Ledger) ByPurpose() map[Purpose]Stat
  • func (l *Ledger) Total() Stat
• type ModelRunner
• type Purpose
• type Runner
  • func NewRunner(prov provider.ModelProvider) *Runner
  • func (r *Runner) Complete(ctx context.Context, p Purpose, req wire.Request) (wire.Response, error)
  • func (r *Runner) Fork() *Runner
  • func (r *Runner) InvalidateModels()
  • func (r *Runner) Ledger() *Ledger
  • func (r *Runner) Provider() provider.ModelProvider
  • func (r *Runner) SetPin(label string)
  • func (r *Runner) SetSession(id string)
  • func (r *Runner) SetVendor(v string)
  • func (r *Runner) Stream(ctx context.Context, p Purpose, req wire.Request, h wire.StreamHandler) (wire.Response, error)
• type Stat

Types

type BackendStat

type BackendStat struct {
	Calls      int
	In, Out    int
	CacheRead  int
	CacheWrite int
	USD        float64 // actual total at the served model's rate card
	LatencyMS  int64   // summed wall-clock; divide by Calls for the mean
	Reasons    map[string]int
	// per-component USD (so /cost can PROVE where the money went — input vs
	// cache-write vs cache-read vs output — instead of hand-waving).
	InUSD, OutUSD, CacheReadUSD, CacheWriteUSD float64
}

BackendStat aggregates per serving backend (cheap | anthropic | openai | …) — where the tokens actually ran and what they cost. USD is the real token bill at the served model's rates: every backend is token-billed (the cheap lane + frontier APIs; the self-hosted/GPU-hours and counterfactual-savings era is gone).

type Ledger

type Ledger struct {
	// contains filtered or unexported fields
}

Ledger is the single, concurrency-safe record of all model usage for a run. Shared across a session and its sub-agents.

func (*Ledger) ByBackend

func (l *Ledger) ByBackend() map[string]BackendStat

ByBackend returns a snapshot of per-backend usage — who actually served the session's calls, at what latency, and what the per-token bill would have been (for /cost backends and the GPU-economics decision).

func (*Ledger) ByPurpose

func (l *Ledger) ByPurpose() map[Purpose]Stat

ByPurpose returns a snapshot of per-purpose usage (for /cost --by-purpose, /route, and learning).

func (*Ledger) Total

func (l *Ledger) Total() Stat

Total returns the aggregate usage + cost across every metered call.

type ModelRunner

type ModelRunner interface {
	Complete(ctx context.Context, p Purpose, req wire.Request) (wire.Response, error)
	Stream(ctx context.Context, p Purpose, req wire.Request, h wire.StreamHandler) (wire.Response, error)
	Ledger() *Ledger
}

ModelRunner is the ONLY interface a model consumer should depend on. Every call carries a Purpose and is metered. (Consumers take this, never provider.ModelProvider.)

type Purpose

type Purpose string

Purpose labels WHY a model call was made, so cost/latency can be attributed per kind of work (and a future router can learn which purposes Haiku handles well).

const (
	MainLoop   Purpose = "main_loop"  // the executive turn loop
	Reflect    Purpose = "reflect"    // plan-mode reflection gate
	Review     Purpose = "review"     // cross-model plan critic (a 2nd cheap model reviews the draft)
	Synth      Purpose = "synth"      // plan synthesis
	Explore    Purpose = "explore"    // read-only scout sub-agent
	Agent      Purpose = "agent"      // first-class delegated sub-agent (agent tool: any tier, read-only or mutating)
	Classify   Purpose = "classify"   // cheap routing/safety classifier
	Predict    Purpose = "predict"    // next-step prediction
	Learn      Purpose = "learn"      // claim/knowledge extraction
	Compact    Purpose = "compact"    // context compaction
	Shrinkwrap Purpose = "shrinkwrap" // one-time MEMCODE.md instruction compression (own concern, not compaction)
	Other      Purpose = "other"      // unlabelled — still counted
)

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner is the metered model-policy gateway over a provider.ModelProvider. Since the all-policy-client-side migration it is also THE routing authority: every call's concrete model is selected here (lane.go semantic ladder + resolve.go physical resolution over the /v1/models control plane), and failed calls recover here (recover.go fallback walk). The backend — gateway, Ollama, anything OpenAI-compatible — just serves what this picks.

func NewRunner

func NewRunner(prov provider.ModelProvider) *Runner

NewRunner wraps a provider with a fresh ledger. Construct ONE at the top level (the cmd boundary) and thread it everywhere — sub-agents share it.

func (*Runner) Complete

func (r *Runner) Complete(ctx context.Context, p Purpose, req wire.Request) (wire.Response, error)

Complete runs a non-streamed call — selection, the recovery walk, and metering (usage, cost, latency, backend).

func (*Runner) Fork

func (r *Runner) Fork() *Runner

Fork returns a NEW Runner that shares this one's Ledger, selection state, and provider connection but is otherwise its own object. Sub-agents Fork the parent's runner instead of sharing the pointer — so what's shared across the main loop and its scouts is passive (the Ledger, the control-plane snapshot), never an executor with per-context state. The pin is inherited: /model pins the whole session, sub-agents included.

func (*Runner) InvalidateModels

func (r *Runner) InvalidateModels()

InvalidateModels drops the control-plane snapshot so the next call refetches — call after /login, /apikeys mutations, or anything else that changes the org's keys/credits/roles. (Billing-class errors invalidate automatically.)

func (*Runner) Ledger

func (r *Runner) Ledger() *Ledger

Ledger exposes the shared usage record.

func (*Runner) Provider

func (r *Runner) Provider() provider.ModelProvider

Provider returns the wrapped provider — ONLY for capability assertions that aren't model calls (e.g. provider.WebSearcher / WebFetcher detection, doctor checks). Never use it to call Complete/Stream directly; that bypasses metering.

func (*Runner) SetPin

func (r *Runner) SetPin(label string)

SetPin ties this Runner to a pinned model label the resolver serves every real request on (Intent.Pin). "" = Automatic (the ladder decides).

func (*Runner) SetSession

func (r *Runner) SetSession(id string)

SetSession ties this Runner to a Session id so every call carries it on the wire (X-Memcode-Session). Called when the owning Session gets its id; forks set their own.

func (*Runner) SetVendor

func (r *Runner) SetVendor(v string)

SetVendor ties this Runner to a per-session strong-tier vendor the ladder resolves tiers within (Intent.Vendor). Called when the Session's vendor changes via /model. The empty string means the configured default (BYOK steering may prefer a keyed vendor).

func (*Runner) Stream

func (r *Runner) Stream(ctx context.Context, p Purpose, req wire.Request, h wire.StreamHandler) (wire.Response, error)

Stream runs a streamed call (live deltas via h) with the same selection + recovery; the emitted-output guard stops the fallback walk the moment any delta reached the user. Falls back to Complete if the provider can't stream.

type Stat

type Stat struct {
	Calls                          int
	In, Out, CacheRead, CacheWrite int
	USD                            float64
}

Stat is an aggregate of usage + estimated cost (total or per-purpose).