engine

Documentation

Overview

Package engine is the inference-engine seam (the EngineDriver). It ships the deterministic, offline building blocks the dispatch chain runs on without a live model or a GPU: a mock engine (the offline fallback) and a cassette record/replay transport, plus the engine-residency adjudicator that denies a tenant-scoped payload routed to a remote engine. The default engine is no longer this mock but the fused in-kernel model (internal/modelengine, id "inkernel").

The live OpenAI-compatible HTTP client lives in internal/agent (HTTPPlanner) — the single outbound /v1/chat/completions seam (one base_url drives local vLLM vs a remote provider). An earlier degenerate HTTPEngine here was a second, never-wired copy of that client that spoke a bespoke `tool=X args=Y` prompt instead of real tool-calling (TICKETS T4); it was deleted so there is exactly one OpenAI client, an invariant pinned by architest's TestSingleOpenAIChatClient.

Index

• Variables
• func CallKey(tool string, args []byte) string
• type CacheEvent
• type CacheEventMetrics
  • func NewCacheEventMetrics() *CacheEventMetrics
  • func (mx *CacheEventMetrics) Snapshot() CacheEventSnapshot
• type CacheEventRecorder
  • func NewCacheEventRecorder() *CacheEventRecorder
  • func (r *CacheEventRecorder) Metrics() *CacheEventMetrics
  • func (r *CacheEventRecorder) Record(ev CacheEvent, opts ...cachemeta.Option) CacheEventResult
  • func (r *CacheEventRecorder) SetSink(fn func(cachemeta.Entry, cachemeta.LookupVerdict))
• type CacheEventResult
  • func (r CacheEventResult) SilentRecompute() bool
• type CacheEventRow
• type CacheEventSnapshot
  • func (s CacheEventSnapshot) Prometheus() string
• type Cassette
  • func LoadCassette(path string) (*Cassette, error)
• type CassetteEngine
  • func NewCassetteEngine(c *Cassette) *CassetteEngine
  • func (e *CassetteEngine) Caps() []abi.Capability
  • func (e *CassetteEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)
• type Mock
  • func (m *Mock) Caps() []abi.Capability
  • func (m *Mock) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)
• type Usage

Variables

var MockEngine = &Mock{}

MockEngine is the registered default (offline-safe). Tests + the bench can also register a CassetteEngine under another id and select it. The live OpenAI-compatible client is internal/agent's HTTPPlanner, not this package.

Functions

func CallKey

func CallKey(tool string, args []byte) string

CallKey is the exported content-address of a (tool, args) pair — used to author cassette entries whose keys the CassetteEngine will match on replay.

Types

type CacheEvent

type CacheEvent struct {
	Direction    cachemeta.KVTransferDirection
	SpanDigest   string
	Tokens       int64
	ModelID      string
	TokenizerID  string
	PositionMode cachemeta.PositionMode
	FromTier     cachemeta.ResidencyTier
	ToTier       cachemeta.ResidencyTier
	Owner        string
	Lease        string
	Outcome      cachemeta.KVTransferOutcome
	FaultReason  string
	BytesMoved   int64
}

CacheEvent is the live-engine residency event a routing/offload adapter feeds the recorder. It is the field-only shape that lowers into cachemeta.KVTransfer; the recorder keeps residency tier and owner separate from the (unseen) payload.

type CacheEventMetrics

type CacheEventMetrics struct {
	// contains filtered or unexported fields
}

CacheEventMetrics is the metric surface over the cache-event stream. It is the "cache events exposed as metrics, not just internal engine counters" half of the §2.2 parity requirement. Counts are keyed by (direction, outcome, to_tier) so a scrape can separate, e.g., a restore fault to remote from an offload ok to dram.

func NewCacheEventMetrics

func NewCacheEventMetrics() *CacheEventMetrics

NewCacheEventMetrics returns an empty, ready metric surface.

func (*CacheEventMetrics) Snapshot

func (mx *CacheEventMetrics) Snapshot() CacheEventSnapshot

Snapshot copies the current counters into a stable, sorted read.

type CacheEventRecorder

type CacheEventRecorder struct {
	// contains filtered or unexported fields
}

CacheEventRecorder is the live-engine cache-event seam. The routing/offload path calls Record for each KV residency transition; the recorder normalizes it into the shared cache-entry stream, derives the typed verdict, folds metrics, and (when set) fans the entry out to an observer sink (e.g. a structured logger). Safe for concurrent use.

func NewCacheEventRecorder

func NewCacheEventRecorder() *CacheEventRecorder

NewCacheEventRecorder returns a recorder with its own metric surface.

func (*CacheEventRecorder) Metrics

func (r *CacheEventRecorder) Metrics() *CacheEventMetrics

Metrics returns the recorder's metric surface for scraping.

func (*CacheEventRecorder) Record

func (r *CacheEventRecorder) Record(ev CacheEvent, opts ...cachemeta.Option) CacheEventResult

Record normalizes one live-engine cache event into the cache-entry stream and returns the entry plus its typed verdict. The verdict makes a failed restore/load a typed MISS/FAULT — the caller must honor that rather than silently recomputing.

func (*CacheEventRecorder) SetSink

func (r *CacheEventRecorder) SetSink(fn func(cachemeta.Entry, cachemeta.LookupVerdict))

SetSink installs an observer called with every normalized (entry, verdict). It is for fan-out only (logging/tracing); it does not change the verdict.

type CacheEventResult

type CacheEventResult struct {
	Entry   cachemeta.Entry
	Verdict cachemeta.LookupVerdict
}

CacheEventResult is what Record returns: the normalized cache-entry (on the kv_transfer plane, in the SAME stream as tool/context entries) plus the typed lookup verdict derived from the event's outcome. A caller that asked for a restore/load reads Verdict and MUST treat a non-Hit as a typed MISS/FAULT — the whole point of this seam is that a failed restore is observable, never a silent recompute.

func (CacheEventResult) SilentRecompute

func (r CacheEventResult) SilentRecompute() bool

SilentRecompute reports whether folding this result as "just recompute it" would hide a real cache fault/miss. It is true for any non-Hit outcome: callers use it to assert the never-silent-recompute rule (a recompute is fine, but only after the miss/fault has been recorded as such, never instead of recording it).

type CacheEventRow

type CacheEventRow struct {
	Direction   string
	Outcome     string
	ToTier      string
	Count       uint64
	BytesMoved  int64
	TokensMoved int64
}

CacheEventRow is one (direction, outcome, to_tier) bucket.

type CacheEventSnapshot

type CacheEventSnapshot struct {
	Events       uint64
	Hits         uint64
	Misses       uint64
	Faults       uint64
	RestoreMiss  uint64
	RestoreFault uint64
	BytesMoved   int64
	TokensMoved  int64
	Rows         []CacheEventRow
}

CacheEventSnapshot is an immutable read of the metric surface, safe to render without holding the lock.

func (CacheEventSnapshot) Prometheus

func (s CacheEventSnapshot) Prometheus() string

Prometheus renders the snapshot as Prometheus exposition text. This is the metric surface a scrape reads to see cache behavior across the live-engine boundary as one stream — not the engine's internal counters. Metric names are namespaced fak_engine_cache_* so they sit alongside the gateway's fak_gateway_*/fak_kernel_*.

type Cassette

type Cassette struct {
	// contains filtered or unexported fields
}

Cassette is a loaded set of recorded interactions. A miss in replay mode is an error (deterministic: a replay must cover its trace).

func LoadCassette

func LoadCassette(path string) (*Cassette, error)

LoadCassette reads a cassette JSON file ({"entries":[...]}).

type CassetteEngine

type CassetteEngine struct {
	// contains filtered or unexported fields
}

CassetteEngine replays a cassette; on miss it errors (StatusError result).

func NewCassetteEngine

func NewCassetteEngine(c *Cassette) *CassetteEngine

func (*CassetteEngine) Caps

func (e *CassetteEngine) Caps() []abi.Capability

func (*CassetteEngine) Complete

func (e *CassetteEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

type Mock

type Mock struct {
	// contains filtered or unexported fields
}

Mock answers any call with a deterministic synthetic result derived from the tool + args, and a simulated token usage proportional to payload size. It lets the whole dispatch chain run with zero network.

func (*Mock) Caps

func (m *Mock) Caps() []abi.Capability

func (*Mock) Complete

func (m *Mock) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

type Usage

type Usage struct {
	InputTokens  int `json:"prompt_tokens"`
	OutputTokens int `json:"completion_tokens"`
	TotalTokens  int `json:"total_tokens"`
}

Usage is the token accounting extracted from a completion (unit 42).