engine

package
v0.40.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 11, 2026 License: Apache-2.0 Imports: 22 Imported by: 0

Documentation

Overview

Package engine is the inference-engine seam (the EngineDriver). It ships the deterministic, offline building blocks the dispatch chain runs on without a live model or a GPU: a mock engine (the offline fallback) and a cassette record/replay transport, plus the engine-residency adjudicator that denies a tenant-scoped payload routed to a remote engine. The default engine is no longer this mock but the fused in-kernel model (internal/modelengine, id "inkernel").

The live OpenAI-compatible HTTP client lives in internal/agent (HTTPPlanner) — the single outbound /v1/chat/completions seam (one base_url drives local vLLM vs a remote provider). An earlier degenerate HTTPEngine here was a second, never-wired copy of that client that spoke a bespoke `tool=X args=Y` prompt instead of real tool-calling (TICKETS T4); it was deleted so there is exactly one OpenAI client, an invariant pinned by architest's TestSingleOpenAIChatClient.

Example (OnDeviceEngine)

Example_onDeviceEngine is the runnable, in-process reference for the on-device topology: wire a phone-class runtime behind the EngineDriver seam, dispatch a tool call to it, and confirm the engine-residency floor keeps a tenant-scoped payload on-box (and denies the same payload bound for a remote engine). Run it with: 

go test ./internal/engine -run Example_onDeviceEngine

ctx := context.Background()

// A deterministic stand-in for a phone-class runtime (llama.cpp / Ollama on-device).
// A real adapter swaps this for the device's local API; the seam is unchanged.
eng := NewOnDeviceEngine(OnDeviceEngineID, OnDeviceRuntimeFunc(
	func(ctx context.Context, prompt string) (string, error) {
		return `{"contacts": 3}`, nil
	}))

// The on-device model proposes a tool call over a tenant-scoped payload. fak
// adjudicates it IN-PROCESS first: on-box → allowed to proceed; remote → denied.
call := odCall("read_contacts", `{"limit":3}`, OnDeviceEngineID, abi.ScopeTenant)
gate := residencyGate{}
onBox := gate.Adjudicate(ctx, call) // the on-device route
leaky := gate.Adjudicate(ctx, odCall("read_contacts", `{"limit":3}`, "litellm/gpt-4o", abi.ScopeTenant))

res, _ := eng.Complete(ctx, call)

fmt.Printf("engine: %s\n", res.Meta["engine"])
fmt.Printf("on-device tenant call denied: %v\n", onBox.Kind == abi.VerdictDeny)
fmt.Printf("remote tenant call denied:    %v\n", leaky.Kind == abi.VerdictDeny)
fmt.Printf("result: %s\n", refBytes(ctx, res.Payload))

Output:
engine: on-device
on-device tenant call denied: false
remote tenant call denied:    true
result: {"contacts": 3}

Index

Examples

Constants

View Source 
const (
	// MetaCacheTenant, MetaCacheAuthority, and MetaCacheFamily are the three fak
	// identity axes folded into the vLLM per-request cache_salt. Tenant is the
	// trust boundary: two tenants with a byte-identical prefix must not share a
	// prefix-cache slot, so tenant is always part of the salt input and a
	// different tenant yields a different salt by construction.
	MetaCacheTenant    = "fak_cache_tenant"
	MetaCacheAuthority = "fak_cache_authority"
	MetaCacheFamily    = "fak_cache_family"

	// MetaTurnPriority is the read-only TurnIntent/session priority. Lower value
	// = earlier, matching vLLM's priority-scheduling convention. It is lowered to
	// the request only when the served engine advertises priority scheduling.
	MetaTurnPriority = "fak_turn_priority"
)

fak-owned request-control meta keys. The gateway/session layer stamps these on a ToolCall so the vLLM adapter can lower fak cache identity and scheduling intent into the request instead of trying to rediscover both below the request boundary. Meta is OPEN, so an unstamped call simply omits the control and the adapter degrades to vLLM's defaults.

View Source 
const DynamoEngineID = "dynamo"

DynamoEngineID is the registered engine id for a Dynamo-managed P/D pool.

View Source 
const LLMDEngineID = "llm-d"

LLMDEngineID is the registered engine id for an llm-d managed serving pool.

View Source 
const LlamaEngineID = "llama.cpp"

LlamaEngineID is the opaque engine label a llama.cpp/llama-server capability is reported under. It matches the "llama.cpp" token the CacheCapability contract names as an example, so the report and the inventory (item 31) agree on engine identity.

View Source 
const OnDeviceEngineID = "on-device"

OnDeviceEngineID is the default engine id the residency floor recognizes as on-box. Register the reference engine under it (or a "on-device:<instance>" variant) so a tenant-scoped payload routed to a phone-native runtime is not denied as a remote leak.

View Source 
const SGLangEngineID = "sglang"

SGLangEngineID is the registered engine id for the SGLang adapter. It is the SAME string enginecache.EngineSGLang uses so the serving adapter and the cache referee agree on engine identity.

View Source 
const VLLMEngineID = "vllm"

VLLMEngineID is the registered engine id for the vLLM V1 adapter.

Variables

View Source 
var DefaultDynamoEngine = NewDynamoEngine(EnvDynamoConfig())

DefaultDynamoEngine is registered under "dynamo". It is inert until configured with FAK_DYNAMO_BASE_URL or replaced in tests via NewDynamoEngine.

View Source 
var DefaultLLMDEngine = NewLLMDEngine(EnvLLMDConfig())

DefaultLLMDEngine is registered under "llm-d". It is inert until configured with FAK_LLMD_BASE_URL / FAK_LLM_D_BASE_URL or replaced in tests via NewLLMDEngine.

View Source 
var DefaultSGLangEngine = NewSGLangEngine(EnvSGLangConfig())

DefaultSGLangEngine is registered under "sglang". It is inert until configured with FAK_SGLANG_BASE_URL (or replaced in tests via NewSGLangEngine).

View Source 
var DefaultVLLMEngine = NewVLLMEngine(EnvVLLMConfig())

DefaultVLLMEngine is registered under "vllm". It is inert until configured with FAK_VLLM_BASE_URL (or replaced in tests via NewVLLMEngine).

View Source 
var MockEngine = &Mock{}

MockEngine is the registered default (offline-safe). Tests + the bench can also register a CassetteEngine under another id and select it. The live OpenAI-compatible client is internal/agent's HTTPPlanner, not this package.

Functions

func CXLPressure added in v0.37.0 

func CXLPressure(residentBytes int64) (pressure float64, capacityBytes int64, known bool)

CXLPressure derives the CXL tier's live fullness and capacity from the box's CPU-less expansion memory (memory-only NUMA nodes, CXL.mem the canonical instance) — the CXL analogue of NUMAFarPressure, with the identical fail-open contract: a box with no expansion node reports known=false and the caller keeps today's behavior.

func CacheTierMemoryClass added in v0.35.0 

func CacheTierMemoryClass(t cachemeta.ResidencyTier) compute.MemoryClass

CacheTierMemoryClass projects a residency tier into the operator-facing memory class used by the capacity/OOM surfaces. HBM is the hot device KV cache; byte-addressable host/far tiers are DDR-cache residency; disk/remote/provider are offload tiers; an unset tier is unknown. This is deliberately a projection over metadata only — it does not move bytes.

func CallKey 

func CallKey(tool string, args []byte) string

CallKey is the exported content-address of a (tool, args) pair — used to author cassette entries whose keys the CassetteEngine will match on replay.

func DeviceHBMPressure added in v0.33.0 

func DeviceHBMPressure(b compute.Backend, residentBytes int64) (pressure float64, capacityBytes int64, known bool)

DeviceHBMPressure derives the HBM tier's live fullness and capacity for backend b from its real device memory — the Plank-3 report->policy conversion, expressed as plain math over compute.DeviceMemoryInfo so it is trivially testable. 

known == false : b cannot probe its capacity (cpu-ref, wasm, any non-DeviceCapacity
                 backend). pressure/capacityBytes are 0 and MUST be ignored — the caller
                 falls back to the profile default (the fail-open contract).
known == true  : total > 0 is the device ceiling (capacityBytes). pressure is the USED
                 fraction in [0,1]:
                   - free known        -> used = total - free  (every consumer of the
                                          device counts, not only fak's tensors)
                   - free FreeUnknown   -> used = residentBytes (the bytes fak tracks
                                          resident in HBM). This is the cuda producer's
                                          case until cudaMemGetInfo is wired (#363
                                          follow-up): total is known, free is not, so
                                          pressure derives from total vs tracked-resident.

residentBytes is clamped into [0,total] so a stale or racy over-count can never push pressure outside [0,1] and trip the placement math into a nonsense decision.

func DiskPressure added in v0.35.0 

func DiskPressure(path string) (pressure float64, capacityBytes int64, known bool)

DiskPressure derives the disk tier's live fullness and capacity for the filesystem at path — the L3 analogue of HostDRAMPressure, expressed as plain math over compute.DiskInfo so it is trivially testable and needs no backend. 

known == false : the path cannot be probed (nonexistent, permission denied, unsupported
                 platform). pressure/capacityBytes are 0 and MUST be ignored — the caller
                 falls back to the profile default (the fail-open contract).
known == true  : total > 0 is the filesystem ceiling (capacityBytes). pressure is the
                 USED fraction in [0,1]: used = total - free.

func HostDRAMPressure added in v0.35.0 

func HostDRAMPressure(residentBytes int64) (pressure float64, capacityBytes int64, known bool)

HostDRAMPressure derives the DRAM tier's live fullness and capacity from the process host's real physical memory — the L2 analogue of DeviceHBMPressure, expressed as plain math over compute.HostSystemMemoryInfo so it is trivially testable and needs no backend. 

known == false : the platform's host-memory probe is unsupported. pressure/capacityBytes
                 are 0 and MUST be ignored — the caller falls back to the profile default
                 (the fail-open contract).
known == true  : total > 0 is the host RAM ceiling (capacityBytes). pressure is the USED
                 fraction in [0,1]:
                   - free known       -> used = total - free  (every consumer of host RAM
                                         counts, not only fak's resident bytes)
                   - free FreeUnknown -> used = residentBytes  (the bytes fak tracks
                                         resident in DRAM, when the probe gives total but
                                         not free)

residentBytes is clamped into [0,total] so a stale or racy over-count can never push pressure outside [0,1] and trip the placement math into a nonsense decision — identical to the HBM wire's clamp.

func NUMAFarPressure added in v0.37.0 

func NUMAFarPressure(residentBytes int64) (pressure float64, capacityBytes int64, known bool)

NUMAFarPressure derives the NUMA-far tier's live fullness and capacity from the far sockets' real memory — the far-memory analogue of HostDRAMPressure, expressed as plain math over compute.NUMAFarMemoryInfo so it is trivially testable and needs no backend. 

known == false : the box has no far NUMA node (single socket), the topology cannot
                 be read, or the platform has no probe. pressure/capacityBytes are
                 0 and MUST be ignored — the caller falls back to the profile
                 default (the fail-open contract).
known == true  : total > 0 is the far sockets' memory ceiling (capacityBytes).
                 pressure is the USED fraction in [0,1], with residentBytes as the
                 fallback when free is unreported — the same shape as the DRAM wire.

func PlanPlacementForDevice added in v0.33.0 

func PlanPlacementForDevice(b compute.Backend, residentBytes int64, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForDevice plans a cachemeta placement against the device that ACTUALLY exists: it derives live HBM pressure + capacity for backend b (DeviceHBMPressure), folds them into req, and calls cachemeta.PlanPlacement. This is the wired path Plank 3 names — the live signal nothing on the serving path computed before; until now only cmd/hwcachedemo injected pressure, by hand.

Fail open: when b cannot probe its capacity (cpu-ref, wasm; known=false) req is used VERBATIM, so a path that worked before is unchanged. residentBytes is fak's tracked-resident HBM byte count — the pressure basis while the cuda producer reports total but not free (#363). req is not mutated: its Pressure and Profiles maps are copied before the HBM override is applied, so a shared request value handed to several backends stays clean.

func PlanPlacementForDeviceAndHost added in v0.35.0 

func PlanPlacementForDeviceAndHost(b compute.Backend, hbmResidentBytes, dramResidentBytes int64, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForDeviceAndHost plans against BOTH live top tiers at once: HBM pressure from the device backend (when present) and DRAM pressure from the host probe. This is the wire a served decode loop on a GPU box wants — a span under HBM pressure may demote to DRAM, and that DRAM target must itself be planned against real host fullness, or the demote lands in a tier that is already full. Each override is independently fail-open: a missing device leaves HBM at its profile default, an unsupported host probe leaves DRAM at its default.

func PlanPlacementForDeviceAtHighWater added in v0.35.0 

func PlanPlacementForDeviceAtHighWater(b compute.Backend, residentBytes int64, targetPressure float64, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForDeviceAtHighWater is PlanPlacementForDevice with an operator high-water mark. A TargetPressure of 0.80 means observed 80% HBM use is presented to cachemeta as "full" pressure, so demotion can happen before the allocator is literally out of memory.

func PlanPlacementForDeviceHostAndDisk added in v0.35.0 

func PlanPlacementForDeviceHostAndDisk(b compute.Backend, hbmResidentBytes, dramResidentBytes int64, diskPath string, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForDeviceHostAndDisk plans against ALL THREE live tiers at once: HBM pressure from the device backend (when present), DRAM pressure from the host probe, and disk pressure from the filesystem probe. This is the full wire a served decode loop on a GPU box wants — a span under HBM pressure may demote to DRAM, then to disk, and each target must itself be planned against real fullness. Each override is independently fail-open.

func PlanPlacementForDisk added in v0.35.0 

func PlanPlacementForDisk(path string, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForDisk plans a cachemeta placement against the filesystem at path that ACTUALLY exists: it derives live disk pressure + capacity (DiskPressure), folds them into req, and calls cachemeta.PlanPlacement. It is the L3 sibling of PlanPlacementForHost — use it when the demote target under consideration is disk and the box has no device backend to probe (the cpu-ref/host-only serve path).

Fail open: when the disk probe fails (known=false) req is used VERBATIM, so a path that worked before is unchanged. req is not mutated: its Pressure and Profiles maps are copied before the disk override is applied, so a shared request value handed to several planners stays clean.

func PlanPlacementForFarMemory added in v0.37.0 

func PlanPlacementForFarMemory(numaResidentBytes, cxlResidentBytes int64, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForFarMemory plans a cachemeta placement against the far memory that actually exists: it derives live NUMA-far and CXL pressure + capacity, folds them into req, and calls cachemeta.PlanPlacement — the far-memory sibling of PlanPlacementForHost and PlanPlacementForDisk. Each fold is independently fail-open (a box with a far socket but no expansion node refines only NUMA-far), and req is not mutated.

func PlanPlacementForHost added in v0.35.0 

func PlanPlacementForHost(residentBytes int64, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForHost plans a cachemeta placement against the HOST that actually exists: it derives live DRAM pressure + capacity (HostDRAMPressure), folds them into req, and calls cachemeta.PlanPlacement. It is the L2 sibling of PlanPlacementForDevice — use it when the demote target under consideration is host DRAM and the box has no device backend to probe (the cpu-ref/host-only serve path).

Fail open: when the host memory probe is unsupported (known=false) req is used VERBATIM, so a path that worked before is unchanged. req is not mutated: its Pressure and Profiles maps are copied before the DRAM override is applied, so a shared request value handed to several planners stays clean.

func PlanPlacementForHostAndDisk added in v0.35.0 

func PlanPlacementForHostAndDisk(dramResidentBytes int64, diskPath string, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForHostAndDisk plans against BOTH live local tiers at once: DRAM pressure from the host probe and disk pressure from the filesystem probe. This is the wire a served decode loop on a cpu-ref box wants — a span under DRAM pressure may demote to disk, and that disk target must itself be planned against real filesystem fullness, or the spill fails. Each override is independently fail-open: an unsupported host probe leaves DRAM at its default, a failed disk probe leaves disk at its default.

func PlanPlacementForLocalLadder added in v0.37.0 

func PlanPlacementForLocalLadder(b compute.Backend, hbmResidentBytes, dramResidentBytes int64, diskPath string, req cachemeta.PlacementRequest) cachemeta.PlacementDecision

PlanPlacementForLocalLadder plans against EVERY live local tier at once — HBM from the device backend, DRAM from the host probe, NUMA-far + CXL from the NUMA topology, and disk from the filesystem probe: the whole in-box relocation ladder (cachemeta.NextColderTier's HBM->DRAM->NUMA-far->CXL->Disk walk), each rung planned against real fullness. This supersedes PlanPlacementForDeviceHostAndDisk as the full wire a served decode loop wants once far tiers can be live too; the far folds pass residentBytes 0 because fak keeps no far-resident counter yet and the topology probe always reports free when it reports at all, so the fallback never engages. Every override remains independently fail-open.

func RegisterResidencyGate added in v0.33.0 

func RegisterResidencyGate()

RegisterResidencyGate installs the engine-residency adjudicator (rank 12) — the same registration init() performs at boot. Exported so a test that resets the ABI registry (abi.ResetForTest) can reinstall the REAL gate rather than a hand-rolled double: e.g. internal/gateway proving a per-call model route written pre-submit is still DENIED for a tenant payload bound for a remote engine. Idempotent (the adjudicator registry is keyed by rank).

Types

type AdapterEngine added in v0.33.0 

type AdapterEngine struct {
	ID   string
	Open UpstreamFactory
}

AdapterEngine is the external-adapter LifecycleEngine. Open is required; ID names the engine for the result payload + Caps.

func NewAdapterEngine added in v0.33.0 

func NewAdapterEngine(id string, open UpstreamFactory) *AdapterEngine

NewAdapterEngine builds an adapter over an upstream factory.

func (*AdapterEngine) Admit added in v0.33.0 

func (a *AdapterEngine) Admit(ctx context.Context, c *abi.ToolCall) (abi.EngineRequest, error)

Admit opens the upstream stream and starts a reader goroutine that pumps its tokens onto the returned handle, honoring ctx (or handle.Cancel) so an abort stops mid-stream and propagates to the upstream via ctx.

func (*AdapterEngine) Caps added in v0.33.0 

func (a *AdapterEngine) Caps() []abi.Capability

Caps advertises the lifecycle seam (negotiable streaming/cancel) plus the adapter's id token.

func (*AdapterEngine) Complete added in v0.33.0 

func (a *AdapterEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

Complete is the one-shot shim so the adapter also satisfies the bare EngineDriver: admit, drain the stream, return the assembled turn.

func (*AdapterEngine) WeightBearing added in v0.37.0 

func (a *AdapterEngine) WeightBearing() bool

WeightBearing declares that an external lifecycle adapter streams model tokens.

type CacheCapability added in v0.38.0 

type CacheCapability struct {
	// Engine is the opaque id of the engine the capability was observed for (e.g.
	// "vllm", "sglang", "llama.cpp"). A label for the report, never an imported type.
	Engine string
	// Verdict is the wire-neutral capability class, a member of the closed
	// CacheVerdict vocabulary. An unrecognized value is not trusted.
	Verdict CacheVerdict
	// Provenance names which plane's evidence backs the verdict, kept separate so a
	// provider rebate counter is never read as a kernel witness (#1490).
	Provenance CacheProvenance
	// Evidence is a short, human-readable anchor for the verdict — a doc reference,
	// control endpoint, or witness id — never the cached bytes.
	Evidence string
	// ColdPathCorrect records that the request's cold path stays correct whether or
	// not this capability's cache behavior fires. It must be true for any ACTIVE
	// verdict (warm / evict / clone); a false value flags an active behavior whose
	// cold-path correctness is not yet witnessed, and a reporter should refuse to
	// trust it (see gateway.ReportEngineCacheCapability).
	ColdPathCorrect bool
}

CacheCapability is the wire-neutral contract for what an upstream inference engine can expose about its cache. It is a value type with NO engine-specific fields and no imports beyond the package's own stdlib usage: "vllm" / "sglang" / "llama.cpp" appear only as an opaque Engine label, never as an imported adapter type. That is what lets the gateway report it without importing engine-specific packages into core (item 32's whole point).

func (CacheCapability) Valid added in v0.38.0 

func (c CacheCapability) Valid() bool

Valid reports whether the capability is well-formed: both its verdict and its provenance are members of their closed vocabularies. It intentionally does NOT fold in ColdPathCorrect — that bit is an explicit provenance signal the reporter acts on (fail-closed) rather than a well-formedness error.

type CacheCapabilityProducer added in v0.38.0 

type CacheCapabilityProducer interface {
	// CacheCapability returns the adapter's wire-neutral cache capability.
	CacheCapability() CacheCapability
}

CacheCapabilityProducer is the seam an engine-specific adapter satisfies to report its cache capability. A reader (the gateway) consumes the capability through THIS interface, so it never imports the adapter's concrete package — the wire-neutral boundary item 32 exists to establish. The #1551–#1553 vLLM / SGLang / llama.cpp adapters implement this behind the seam; core imports the interface, not them.

type CacheEvent 

type CacheEvent struct {
	Direction    cachemeta.KVTransferDirection
	SpanDigest   string
	Tokens       int64
	ModelID      string
	TokenizerID  string
	PositionMode cachemeta.PositionMode
	FromTier     cachemeta.ResidencyTier
	ToTier       cachemeta.ResidencyTier
	Owner        string
	Lease        string
	Outcome      cachemeta.KVTransferOutcome
	FaultReason  string
	BytesMoved   int64
}

CacheEvent is the live-engine residency event a routing/offload adapter feeds the recorder. It is the field-only shape that lowers into cachemeta.KVTransfer; the recorder keeps residency tier and owner separate from the (unseen) payload.

type CacheEventMetrics 

type CacheEventMetrics struct {
	// contains filtered or unexported fields
}

CacheEventMetrics is the metric surface over the cache-event stream. It is the "cache events exposed as metrics, not just internal engine counters" half of the §2.2 parity requirement. Counts are keyed by (direction, outcome, to_tier) so a scrape can separate, e.g., a restore fault to remote from an offload ok to dram.

func NewCacheEventMetrics 

func NewCacheEventMetrics() *CacheEventMetrics

NewCacheEventMetrics returns an empty, ready metric surface.

func (*CacheEventMetrics) Snapshot 

func (mx *CacheEventMetrics) Snapshot() CacheEventSnapshot

Snapshot copies the current counters into a stable, sorted read.

type CacheEventRecorder 

type CacheEventRecorder struct {
	// contains filtered or unexported fields
}

CacheEventRecorder is the live-engine cache-event seam. The routing/offload path calls Record for each KV residency transition; the recorder normalizes it into the shared cache-entry stream, derives the typed verdict, folds metrics, and (when set) fans the entry out to an observer sink (e.g. a structured logger). Safe for concurrent use.

func NewCacheEventRecorder 

func NewCacheEventRecorder() *CacheEventRecorder

NewCacheEventRecorder returns a recorder with its own metric surface.

func (*CacheEventRecorder) Metrics 

func (r *CacheEventRecorder) Metrics() *CacheEventMetrics

Metrics returns the recorder's metric surface for scraping.

func (*CacheEventRecorder) Record 

func (r *CacheEventRecorder) Record(ev CacheEvent, opts ...cachemeta.Option) CacheEventResult

Record normalizes one live-engine cache event into the cache-entry stream and returns the entry plus its typed verdict. The verdict makes a failed restore/load a typed MISS/FAULT — the caller must honor that rather than silently recomputing.

func (*CacheEventRecorder) SetSink 

func (r *CacheEventRecorder) SetSink(fn func(cachemeta.Entry, cachemeta.LookupVerdict))

SetSink installs an observer called with every normalized (entry, verdict). It is for fan-out only (logging/tracing); it does not change the verdict.

type CacheEventResult 

type CacheEventResult struct {
	Entry   cachemeta.Entry
	Verdict cachemeta.LookupVerdict
}

CacheEventResult is what Record returns: the normalized cache-entry (on the kv_transfer plane, in the SAME stream as tool/context entries) plus the typed lookup verdict derived from the event's outcome. A caller that asked for a restore/load reads Verdict and MUST treat a non-Hit as a typed MISS/FAULT — the whole point of this seam is that a failed restore is observable, never a silent recompute.

func RecordVLLMKVEventBatch added in v0.35.0 

func RecordVLLMKVEventBatch(worker, model, tokenizer string, idx *PrefixResidencyIndex, rec *CacheEventRecorder, batch VLLMKVEventBatch) []CacheEventResult

RecordVLLMKVEventBatch is the pure lowering function for vLLM KV events.

func (CacheEventResult) SilentRecompute 

func (r CacheEventResult) SilentRecompute() bool

SilentRecompute reports whether folding this result as "just recompute it" would hide a real cache fault/miss. It is true for any non-Hit outcome: callers use it to assert the never-silent-recompute rule (a recompute is fine, but only after the miss/fault has been recorded as such, never instead of recording it).

type CacheEventRow 

type CacheEventRow struct {
	Direction   string
	Outcome     string
	ToTier      string
	MemoryClass string
	Count       uint64
	BytesMoved  int64
	TokensMoved int64
}

CacheEventRow is one (direction, outcome, to_tier, memory_class) bucket.

type CacheEventSnapshot 

type CacheEventSnapshot struct {
	Events       uint64
	Hits         uint64
	Misses       uint64
	Faults       uint64
	RestoreMiss  uint64
	RestoreFault uint64
	BytesMoved   int64
	TokensMoved  int64
	Rows         []CacheEventRow

	// KeysCapped is true once byKey has hit maxCacheEventKeys — further distinct
	// (direction, outcome, to_tier, memory_class) keys fold into the overflow
	// bucket below rather than growing Rows. OverflowEvents/Bytes/Tokens are that
	// bucket's totals, so the per-key breakdown never silently loses mass.
	KeysCapped          bool
	OverflowEvents      uint64
	OverflowBytesMoved  int64
	OverflowTokensMoved int64
}

CacheEventSnapshot is an immutable read of the metric surface, safe to render without holding the lock.

func (CacheEventSnapshot) Prometheus 

func (s CacheEventSnapshot) Prometheus() string

Prometheus renders the snapshot as Prometheus exposition text. This is the metric surface a scrape reads to see cache behavior across the live-engine boundary as one stream — not the engine's internal counters. Metric names are namespaced fak_engine_cache_* so they sit alongside the gateway's fak_gateway_*/fak_kernel_*.

type CacheProvenance added in v0.38.0 

type CacheProvenance string

CacheProvenance names the plane whose evidence backs a CacheCapability verdict. Epic #1490 requires these stay SEPARATE in any reported value: an OBSERVED provider counter is not a WITNESSED kernel event, and a FORECAST is neither. Carrying provenance in the contract stops a report from collapsing different trust levels into one label. 

const (
	// ProvenanceKernel: a fak in-kernel WITNESSED event backs the verdict. Also the
	// gateway's own default when it reports from its own knowledge.
	ProvenanceKernel CacheProvenance = "kernel"
	// ProvenanceProvider: an OBSERVED provider / engine counter backs the verdict.
	ProvenanceProvider CacheProvenance = "provider"
	// ProvenanceContext: O(1) context / session-memory evidence backs the verdict.
	ProvenanceContext CacheProvenance = "context"
	// ProvenanceForecast: a FORECAST / synthetic estimate backs the verdict — never a
	// measured fact.
	ProvenanceForecast CacheProvenance = "forecast"
)

func CacheProvenances added in v0.38.0 

func CacheProvenances() []CacheProvenance

CacheProvenances returns the closed provenance vocabulary in a stable order.

func (CacheProvenance) Valid added in v0.38.0 

func (p CacheProvenance) Valid() bool

Valid reports whether p is a member of the closed provenance vocabulary.

type CacheRefusal added in v0.38.0 

type CacheRefusal string

CacheRefusal is the CLOSED vocabulary of named reasons an active cache operation was refused. The zero value (RefusalNone) is the ONLY admitted value — so a caller that forgets to check, or a future member added to this enum, fails closed by construction rather than reading as an admit. 

const (
	// RefusalNone is the zero value: the active cache operation was ADMITTED. It is the
	// only non-refusal member of the vocabulary.
	RefusalNone CacheRefusal = ""
	// RefusalOperationNotActive: the requested operation is not an ACTIVE cache verdict
	// (warm / exact-evict / prefix-clone). Passive observation, paged-KV structure and
	// unknown are not operations that can be admitted here — this names caller misuse
	// rather than an engine shortcoming.
	RefusalOperationNotActive CacheRefusal = "operation-not-active"
	// RefusalCapabilityMalformed: the capability is not well-formed — its verdict or its
	// provenance falls outside the closed vocabulary, so nothing about it is trusted.
	RefusalCapabilityMalformed CacheRefusal = "capability-malformed"
	// RefusalCapabilityUnknown: the engine/provider capability is not established
	// (CacheUnknown). This is the issue's headline refusal — an UNKNOWN capability must
	// never fall through to an optimistic claim that the active primitive is available.
	RefusalCapabilityUnknown CacheRefusal = "capability-unknown"
	// RefusalOperationUnsupported: the capability is established, but for a DIFFERENT
	// class than the requested operation (e.g. an exact-evict engine asked to
	// active-warm). The active path is unsupported on this engine.
	RefusalOperationUnsupported CacheRefusal = "operation-unsupported"
	// RefusalColdPathUnwitnessed: the capability claims an active behavior whose cold
	// path has not been witnessed correct (ColdPathCorrect == false). Cold-path
	// correctness stays explicit for active behavior; an unwitnessed one is refused.
	RefusalColdPathUnwitnessed CacheRefusal = "cold-path-unwitnessed"
	// RefusalForecastProvenance: the capability is backed only by a FORECAST / synthetic
	// estimate, which is never a measured fact. Acting on it would be exactly the
	// optimistic claim #1490 removes, so an active operation demands a measured plane
	// (kernel / provider / context). Provenance stays separate from the verdict: this
	// refuses the ACT, it does not rewrite the reported capability.
	RefusalForecastProvenance CacheRefusal = "forecast-provenance"
)

func AdmitActiveCache added in v0.38.0 

func AdmitActiveCache(cc CacheCapability, op CacheVerdict) CacheRefusal

AdmitActiveCache is the single fail-closed admission point for an ACTIVE cache operation (op) against an engine's reported capability (cc).

It returns RefusalNone ONLY when every honesty condition holds:

  • op is an ACTIVE verdict (warm / exact-evict / prefix-clone);
  • cc is well-formed (verdict and provenance both in their closed vocabularies);
  • cc's capability is established (not CacheUnknown);
  • cc's capability is for exactly the requested operation — a CacheCapability carries a single verdict, so an engine that supports two active classes reports two capabilities;
  • cc's cold path is witnessed correct;
  • cc is backed by a MEASURED plane, not a forecast.

Otherwise it returns a NAMED reason from the closed CacheRefusal vocabulary. It never panics and never returns an optimistic admit for an unknown or unsupported capability: the caller's correct response to any refusal is the COLD path (send the full required context), which stays correct whether or not any cache behavior fires.

func CacheRefusals added in v0.38.0 

func CacheRefusals() []CacheRefusal

CacheRefusals returns the closed refusal vocabulary in a stable order. A report or conformance test iterates it to prove the enum admits ONLY these members.

func (CacheRefusal) Refused added in v0.38.0 

func (r CacheRefusal) Refused() bool

Refused reports whether r names a refusal (i.e. the operation was NOT admitted). Anything other than RefusalNone is a refusal, so an unrecognized value read off a wire is treated as refused rather than admitted.

func (CacheRefusal) Valid added in v0.38.0 

func (r CacheRefusal) Valid() bool

Valid reports whether r is a member of the closed refusal vocabulary.

type CacheVerdict added in v0.38.0 

type CacheVerdict string

CacheVerdict is the CLOSED vocabulary of engine cache capabilities — the only wire-neutral labels a report may use. It matches the capability-inventory terms (item 31 / #1549) so the contract and the inventory never drift, and a new engine cannot smuggle in a bespoke class. The zero value is the empty string, which is NOT a member of the vocabulary (Valid() == false): an unset verdict is fail-closed to CacheUnknown by a reporter (see gateway.ReportEngineCacheCapability), never read as a fabricated positive. CacheUnknown is the explicit "capability not established" label a producer sets on purpose. 

const (
	// CacheUnknown is the zero value: the engine's cache capability is not
	// established. The safe default — never inferred as a positive.
	CacheUnknown CacheVerdict = "unknown"
	// CachePassiveObserve: the engine exposes cache SYMPTOMS the gateway can observe
	// (usage counters, cached-token telemetry) but no control surface to act on.
	CachePassiveObserve CacheVerdict = "passive-observe"
	// CacheActiveWarm: the engine can be actively warmed — prefill a named prefix on
	// demand so a later request hits it.
	CacheActiveWarm CacheVerdict = "active-warm"
	// CacheExactEvict: the engine can evict an EXACT named span, not merely flush the
	// whole prefix/radix cache.
	CacheExactEvict CacheVerdict = "exact-evict"
	// CachePrefixClone: the engine can clone/fork a cached prefix for reuse across
	// sessions.
	CachePrefixClone CacheVerdict = "prefix-clone"
	// CachePagedKV: the engine exposes paged-KV behavior — block-addressable KV the
	// gateway can reason about.
	CachePagedKV CacheVerdict = "paged-kv"
)

func CacheVerdicts added in v0.38.0 

func CacheVerdicts() []CacheVerdict

CacheVerdicts returns the closed verdict vocabulary in a stable order. A report or conformance test iterates it to prove the enum admits ONLY these members.

func (CacheVerdict) Active added in v0.38.0 

func (v CacheVerdict) Active() bool

Active reports whether the verdict describes ACTIVE cache behavior (warm / exact evict / prefix clone) — the class for which cold-path correctness must be witnessed (ColdPathCorrect). Passive observation, paged-KV structure, and unknown are not active behaviors: they change nothing a request can come to depend on.

func (CacheVerdict) Valid added in v0.38.0 

func (v CacheVerdict) Valid() bool

Valid reports whether v is a member of the closed vocabulary.

type CapacityAdapter added in v0.33.0 

type CapacityAdapter struct {
	KV       abi.KVBackend
	Recorder *CacheEventRecorder
}

CapacityAdapter is the engine adapter that executes a cachemeta placement decision against the kernel-owned KV cache. It drives the demote/spill/evict family — the actions that DROP a span from a hot tier — through the live cache (KV) and records each transition through the cache-event stream (Recorder). PlanPlacement decided the move; this adapter MAKES the move. A nil Recorder is allowed: the move still executes, it is simply not folded into the metric stream.

func (*CapacityAdapter) Execute added in v0.33.0 

func (a *CapacityAdapter) Execute(ctx context.Context, mv PlacementMove) (PlacementResult, error)

Execute carries out a placement move against the live KV cache. For the demote/spill family it:

  1. STAGES the span to the colder tier (KV.StageSpan), addressed by digest so the disaggregated direction survives. The live copy is dropped only on a confirmed OK — a typed MISS/FAULT retains the span and records the fault (fail-safe).
  2. EVICTS the span from the live KV tier (KV.Evict, the re-RoPE/renumber primitive), reclaiming room in the hot tier.
  3. RECORDS the offload as a typed CacheEvent so the move enters the cache-entry stream — a staging fault is a FAULT(residency_fault), never silent.

An evict (no colder tier had room) skips staging and drops the span outright — it is the recompute-on-demand path. A promote (KVRestore) is the reverse direction and is NOT executed here; Execute returns Applied=false so a caller routes it to the restore path. A keep is a no-op. A nil KV backend is a typed error rather than a nil-deref.

type CapacityPressureCandidate added in v0.35.0 

type CapacityPressureCandidate struct {
	Request      cachemeta.PlacementRequest
	Move         PlacementMove
	ReclaimBytes int64
}

CapacityPressureCandidate is one live KV span the pressure sweep may move out of HBM. Request carries the span's placement economics; Move carries its executable identity in the live KV backend. ReclaimBytes is the HBM byte estimate to subtract after a successful move; when unset, Request.SizeBytes is used.

type CapacityPressureMove added in v0.35.0 

type CapacityPressureMove struct {
	Index    int
	Decision cachemeta.PlacementDecision
	Result   PlacementResult
}

CapacityPressureMove records the decision and execution result for one candidate the sweep attempted.

type CapacityPressureResult added in v0.35.0 

type CapacityPressureResult struct {
	Known           bool
	CapacityBytes   int64
	TargetPressure  float64
	InitialPressure float64
	FinalPressure   float64
	ReclaimedBytes  int64
	AppliedMoves    int
	Faults          int
	Moves           []CapacityPressureMove
}

CapacityPressureResult is the typed outcome of one pressure sweep.

func RunCapacityPressureSweep added in v0.35.0 

func RunCapacityPressureSweep(ctx context.Context, cfg CapacityPressureSweep) (CapacityPressureResult, error)

RunCapacityPressureSweep relieves HBM pressure by planning and executing moves for candidate KV spans until the estimated pressure drops below TargetPressure, the move cap is reached, or candidates are exhausted. Unknown capacity is a clean no-op. Staging faults are recorded in the result but do not abort the sweep, so a single bad colder tier cannot hide pressure on the remaining candidates.

type CapacityPressureSweep added in v0.35.0 

type CapacityPressureSweep struct {
	Backend        compute.Backend
	Adapter        *CapacityAdapter
	ResidentBytes  int64
	TargetPressure float64
	MaxMoves       int
	Candidates     []CapacityPressureCandidate

	// DRAMPressure, DRAMCapacityBytes, and DRAMKnown carry the host's live L2/DRAM fullness as
	// probed by the CALLER (the served loop's HostDRAMPressure — see capacity_dram.go), threaded
	// in so the demote TARGET is planned against real host RAM rather than the assumed-empty
	// profile default. Without them the sweep was hardware-aware at its top rung (HBM, via
	// Backend) but BLIND one tier below: a span under HBM pressure that PlanPlacement demotes
	// INTO DRAM was staged there even on a box whose RAM is already full, where the honest move
	// is to skip DRAM for a colder tier with room. This is the "probe DeviceHBMPressure AND
	// HostDRAMPressure" half of issue #1073's wire on the executor side (PlanPlacementForDeviceAndHost
	// is the planless sibling of the same fold).
	//
	// DRAMKnown gates the fold and is the fail-open contract: false (the default — an unsupported
	// host probe, or a caller that does not probe DRAM) folds nothing, so the sweep plans exactly
	// as it did before these fields existed. The probe is the caller's; the sweep only ACTS on it,
	// staying a pure, deterministic function of its config.
	DRAMPressure      float64
	DRAMCapacityBytes int64
	DRAMKnown         bool
}

CapacityPressureSweep configures one bounded pressure-relief pass. TargetPressure is the desired HBM high-water mark in (0,1]; values outside that range default to 1.0, preserving the older "only full means pressure" behavior. MaxMoves <= 0 means no explicit move cap beyond the candidate list.

type Cassette 

type Cassette struct {
	// contains filtered or unexported fields
}

Cassette is a loaded set of recorded interactions. A miss in replay mode is an error (deterministic: a replay must cover its trace).

func LoadCassette 

func LoadCassette(path string) (*Cassette, error)

LoadCassette reads a cassette JSON file ({"entries":[...]}).

func NewCassette added in v0.37.0 

func NewCassette(entries []CassetteEntry) *Cassette

NewCassette builds a Cassette from in-memory entries (the counterpart to LoadCassette's from-disk path). Each entry is keyed by CallKey(Tool, Args), so a CassetteEngine built from it replays the REAL usage the entry carries for that exact (tool, args) pair — this is what lets `fak ablate --from-session` report actual provider token usage instead of the mock engine's payload-size synthesis. A missing Response is filled with a minimal synthesized echo so Complete always has a payload to return.

type CassetteEngine 

type CassetteEngine struct {
	// contains filtered or unexported fields
}

CassetteEngine replays a cassette; on miss it errors (StatusError result).

func NewCassetteEngine 

func NewCassetteEngine(c *Cassette) *CassetteEngine

NewCassetteEngine wraps a loaded Cassette in an engine that replays its recorded (tool, args) -> response interactions.

func (*CassetteEngine) Caps 

func (e *CassetteEngine) Caps() []abi.Capability

Caps reports the engine's capabilities; the cassette replayer declares none.

func (*CassetteEngine) Complete 

func (e *CassetteEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

type CassetteEntry added in v0.37.0 

type CassetteEntry struct {
	Tool     string
	Args     []byte
	Response json.RawMessage // raw result payload bytes ("" => a synthesized ok:true echo)
	Usage    Usage
	Meta     map[string]string
}

CassetteEntry is the exported shape of one recorded (tool, args) -> usage interaction, for a caller that builds a Cassette IN-PROCESS (e.g. from a captured guard/serve session) rather than loading one from a JSON file on disk. Key is computed by NewCassette from Tool+Args via CallKey, so a caller never hand-rolls the content hash.

type DynamoConfig added in v0.35.0 

type DynamoConfig struct {
	BaseURL    string
	Model      string
	APIKey     string
	WorkerID   string
	MetricsURL string
	Client     *http.Client
}

DynamoConfig wires one Dynamo frontend through public surfaces only: the OpenAI-compatible /v1 routes for generation and Prometheus /metrics for P/D worker observation.

func EnvDynamoConfig added in v0.35.0 

func EnvDynamoConfig() DynamoConfig

EnvDynamoConfig returns the default Dynamo driver configuration. FAK_DYNAMO_BASE_URL should point at Dynamo's OpenAI-compatible frontend root, usually http://host:port/v1.

type DynamoEngine added in v0.35.0 

type DynamoEngine struct {
	// contains filtered or unexported fields
}

DynamoEngine is an abi.EngineDriver/LifecycleEngine adapter for a Dynamo-managed P/D serving pool.

func NewDynamoEngine added in v0.35.0 

func NewDynamoEngine(cfg DynamoConfig) *DynamoEngine

NewDynamoEngine builds a Dynamo driver over public Dynamo surfaces.

func (*DynamoEngine) Admit added in v0.35.0 

func (e *DynamoEngine) Admit(ctx context.Context, c *abi.ToolCall) (abi.EngineRequest, error)

Admit submits one request to Dynamo's OpenAI-compatible frontend with stream=true and returns a live request handle. Dynamo owns routing across its P/D pool behind that frontend; fak stays in front as the governance plane.

func (*DynamoEngine) Caps added in v0.35.0 

func (e *DynamoEngine) Caps() []abi.Capability

Caps advertises Dynamo ride-mode support: OpenAI-compatible frontend dispatch, lifecycle streaming, Dynamo metrics normalization, and the honest whole-prefix governance boundary for a ridden engine whose exact KV span API is not exposed.

func (*DynamoEngine) Complete added in v0.35.0 

func (e *DynamoEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

Complete drains the live stream and returns the assembled result.

func (*DynamoEngine) ScrapeServingMetrics added in v0.35.0 

func (e *DynamoEngine) ScrapeServingMetrics(ctx context.Context) (DynamoServingMetrics, error)

ScrapeServingMetrics reads Dynamo's Prometheus endpoint and returns one normalized fak_serving_* row per observed worker.

func (*DynamoEngine) WeightBearing added in v0.37.0 

func (e *DynamoEngine) WeightBearing() bool

WeightBearing declares that Dynamo dispatch runs a model-forward, not a deterministic tool.

type DynamoServingMetrics added in v0.35.0 

type DynamoServingMetrics struct {
	Rows []ServingMetricsSnapshot
}

DynamoServingMetrics is Dynamo's multi-worker view normalized into fak's L2 serving schema.

func ParseDynamoPrometheus added in v0.35.0 

func ParseDynamoPrometheus(defaultWorker, text string) DynamoServingMetrics

ParseDynamoPrometheus maps Dynamo public Prometheus metrics into fak_serving_*. It accepts current Dynamo frontend/worker names and a few stable label aliases so the adapter is tolerant of deployment relabeling while still ignoring unknown Dynamo-only metrics.

func (DynamoServingMetrics) Prometheus added in v0.35.0 

func (m DynamoServingMetrics) Prometheus() string

type LLMDConfig added in v0.37.0 

type LLMDConfig struct {
	BaseURL    string
	Model      string
	APIKey     string
	WorkerID   string
	MetricsURL string
	Client     *http.Client

	CacheRecorder *CacheEventRecorder
	Residency     *PrefixResidencyIndex
	KVEvents      VLLMKVEventSource
}

LLMDConfig wires one llm-d OpenAI-compatible frontend through public surfaces only: /v1 routes for generation, optional Prometheus /metrics for vLLM worker observation, and optional decoded vLLM KV events for residency.

func EnvLLMDConfig added in v0.37.0 

func EnvLLMDConfig() LLMDConfig

EnvLLMDConfig returns the default llm-d driver configuration. FAK_LLMD_BASE_URL should point at the llm-d Gateway API frontend's OpenAI-compatible /v1 root. FAK_LLM_D_* aliases are accepted because the upstream project name includes a hyphen, while environment variables cannot.

type LLMDEngine added in v0.37.0 

type LLMDEngine struct {
	// contains filtered or unexported fields
}

LLMDEngine is an abi.EngineDriver/LifecycleEngine adapter for an llm-d managed fleet. It reuses the vLLM-compatible request and metrics lowerings because llm-d frontends route OpenAI-compatible requests to vLLM workers.

func NewLLMDEngine added in v0.37.0 

func NewLLMDEngine(cfg LLMDConfig) *LLMDEngine

NewLLMDEngine builds an llm-d driver over public llm-d/vLLM-compatible surfaces.

func (*LLMDEngine) Admit added in v0.37.0 

func (e *LLMDEngine) Admit(ctx context.Context, c *abi.ToolCall) (abi.EngineRequest, error)

Admit submits one request to the llm-d OpenAI-compatible frontend with stream=true and returns a live request handle. llm-d owns placement behind that frontend; fak records the served result under engine="llm-d".

func (*LLMDEngine) Caps added in v0.37.0 

func (e *LLMDEngine) Caps() []abi.Capability

Caps advertises llm-d ride-mode support: OpenAI-compatible frontend dispatch, lifecycle streaming, Gateway API / EPP managed routing, vLLM-worker metrics, KV event ingestion, and the honest whole-prefix governance boundary for an external engine whose exact KV span API is not exposed.

func (*LLMDEngine) Complete added in v0.37.0 

func (e *LLMDEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

Complete drains the live stream and returns the assembled result.

func (*LLMDEngine) RunKVEventSubscription added in v0.37.0 

func (e *LLMDEngine) RunKVEventSubscription(ctx context.Context) error

RunKVEventSubscription consumes decoded vLLM KV-event batches from the llm-d worker plane when a deployment supplies that bridge.

func (*LLMDEngine) ScrapeServingMetrics added in v0.37.0 

func (e *LLMDEngine) ScrapeServingMetrics(ctx context.Context) (ServingMetricsSnapshot, error)

ScrapeServingMetrics reads the llm-d/vLLM Prometheus endpoint and normalizes it under engine="llm-d" in fak's serving schema.

func (*LLMDEngine) WeightBearing added in v0.37.0 

func (e *LLMDEngine) WeightBearing() bool

WeightBearing declares that llm-d dispatch runs a model-forward, not a deterministic tool.

type LlamaCacheObserver added in v0.38.0 

type LlamaCacheObserver struct {
	// contains filtered or unexported fields
}

LlamaCacheObserver reads llama-server's session-cache signal and reports it as a wire-neutral engine.CacheCapability. It satisfies CacheCapabilityProducer behind the item-32 seam, so the gateway can report a llama-backed session's cache capability without importing this package.

func NewLlamaCacheObserver added in v0.38.0 

func NewLlamaCacheObserver(cfg LlamaConfig) *LlamaCacheObserver

NewLlamaCacheObserver builds an observer over a llama-server's public /slots surface.

func (*LlamaCacheObserver) CacheCapability added in v0.38.0 

func (o *LlamaCacheObserver) CacheCapability() CacheCapability

CacheCapability satisfies engine.CacheCapabilityProducer. Before Observe has resolved a signal it FAILS CLOSED to the unknown/no-evidence label rather than inferring a positive — the same fail-closed default the contract's zero verdict has.

func (*LlamaCacheObserver) Observe added in v0.38.0 

func (o *LlamaCacheObserver) Observe(ctx context.Context) (LlamaSessionCache, error)

Observe reads the /slots session-cache signal once and caches it for CacheCapability. A transport error is returned; a disabled/empty endpoint is not an error — it resolves to the passive/no-evidence signal. Safe to call again to refresh.

type LlamaConfig added in v0.38.0 

type LlamaConfig struct {
	// BaseURL is the llama-server HTTP root (host:port llama-server binds). /slots is
	// derived from it when SlotsURL is empty.
	BaseURL string
	// SlotsURL overrides the derived <BaseURL>/slots endpoint when a proxy/bridge
	// exposes the slots telemetry at a non-default path.
	SlotsURL string
	// APIKey is an optional bearer token for a fronted llama-server.
	APIKey string
	// Client is an optional HTTP client; nil falls back to the shared default.
	Client *http.Client
}

LlamaConfig wires one llama.cpp/llama-server instance through its PUBLIC read-only surface only — the /slots telemetry endpoint. It deliberately adds no generation path: this adapter observes a cache signal, it does not serve.

func EnvLlamaConfig added in v0.38.0 

func EnvLlamaConfig() LlamaConfig

EnvLlamaConfig returns the default llama observation configuration from the FAK_LLAMA_* environment. FAK_LLAMA_BASE_URL should point at the llama-server HTTP root; FAK_LLAMA_SLOTS_URL overrides the derived /slots path.

type LlamaSessionCache added in v0.38.0 

type LlamaSessionCache struct {
	// Present is true when /slots exposed an observable session-cache symptom: at
	// least one slot carrying a prompt-cache accounting field. False when the endpoint
	// is disabled, empty, or exposes no such field.
	Present bool
	// Slots is the number of slots the response covered.
	Slots int
	// Fields names the prompt-cache accounting fields observed on the slots (e.g.
	// "cache_tokens", "n_past"), sorted and de-duplicated. It is the human anchor for
	// the passive-observe verdict, not a reuse count.
	Fields []string
	// Note explains a signal-ABSENT observation (endpoint disabled, empty body, no
	// accounting field). Empty when Present.
	Note string
}

LlamaSessionCache is the decoded session-cache signal read from a llama-server /slots response. It records only WHETHER a session-cache symptom surface is exposed and which accounting fields carried it — never a claimed reuse figure, so the passive/no-evidence honesty boundary cannot be crossed by accident.

func ObserveLlamaSessionCache added in v0.38.0 

func ObserveLlamaSessionCache(ctx context.Context, client *http.Client, slotsURL, apiKey string) (LlamaSessionCache, error)

ObserveLlamaSessionCache fetches one /slots snapshot and decodes its session-cache signal. A transport error is returned to the caller; a NON-200 status (llama-server answers 501 when --slots is disabled) is NOT an error — it resolves to the passive/no-evidence signal, because a disabled endpoint is a legitimate "engine exposes no cache surface" observation rather than a failed read.

func (LlamaSessionCache) Capability added in v0.38.0 

func (s LlamaSessionCache) Capability() CacheCapability

Capability maps the decoded signal onto the wire-neutral engine.CacheCapability contract. A present symptom surface is CachePassiveObserve; an absent one is the fail-closed CacheUnknown with an explicit passive/no-evidence Evidence string. Both carry ProvenanceProvider (an observed provider counter, not a kernel witness) and ColdPathCorrect true (observation never changes the request's cold path).

type Mock 

type Mock struct {
	// contains filtered or unexported fields
}

Mock answers any call with a deterministic synthetic result derived from the tool + args, and a simulated token usage proportional to payload size. It lets the whole dispatch chain run with zero network.

func (*Mock) Caps 

func (m *Mock) Caps() []abi.Capability

Caps reports the engine's capabilities; the mock declares none.

func (*Mock) Complete 

func (m *Mock) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

type OnDeviceEngine added in v0.35.0 

type OnDeviceEngine struct {
	ID      string
	Runtime OnDeviceRuntime
}

OnDeviceEngine is the reference EngineDriver wrapping a phone-class on-device runtime. It satisfies the bare abi.EngineDriver (Complete + Caps); the kernel dispatches an admitted tool call to it exactly as it would the in-kernel or mock engine. ID names the runtime for the result meta + Caps and keys residency; Runtime is the pluggable phone-class backend.

func NewOnDeviceEngine added in v0.35.0 

func NewOnDeviceEngine(id string, rt OnDeviceRuntime) *OnDeviceEngine

NewOnDeviceEngine builds the reference on-device engine over a runtime. An empty id defaults to OnDeviceEngineID so the residency floor recognizes the route as on-box; pass a "on-device:<instance>" form to register several devices.

func (*OnDeviceEngine) Caps added in v0.35.0 

func (e *OnDeviceEngine) Caps() []abi.Capability

Caps advertises the on-device engine family plus this instance's id token. It does NOT advertise engine.openai: this adapter speaks to a local runtime, not the OpenAI HTTP wire (which has exactly one client, in internal/agent).

func (*OnDeviceEngine) Complete added in v0.35.0 

func (e *OnDeviceEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

Complete runs the tool call against the on-device runtime and returns the assembled result. It FAILS CLOSED: a nil runtime or a runtime error yields a StatusError result (never a panic, never a silent StatusOK), so the dispatch chain sees the miss rather than treating a wedged device as a successful empty answer.

func (*OnDeviceEngine) WeightBearing added in v0.37.0 

func (e *OnDeviceEngine) WeightBearing() bool

WeightBearing declares that the on-device runtime runs a local model-forward.

type OnDeviceRuntime added in v0.35.0 

type OnDeviceRuntime interface {
	Generate(ctx context.Context, prompt string) (string, error)
}

OnDeviceRuntime is the minimal completion surface a phone-class on-device engine exposes: given the prompt for one tool call, return the generated text. It is the single-shot counterpart of the server-class UpstreamStream (lifecycle_adapter.go) — a phone runtime answers a whole turn locally rather than streaming an async token API. A real adapter implements it over llama.cpp (CGo or a llama-server child) or Ollama's local daemon; ctx bounds a stall so a wedged device surfaces as an error, not a hang.

type OnDeviceRuntimeFunc added in v0.35.0 

type OnDeviceRuntimeFunc func(ctx context.Context, prompt string) (string, error)

OnDeviceRuntimeFunc adapts a plain function to an OnDeviceRuntime, so a reference or a test can wire a runtime without declaring a named type.

func (OnDeviceRuntimeFunc) Generate added in v0.35.0 

func (f OnDeviceRuntimeFunc) Generate(ctx context.Context, prompt string) (string, error)

Generate calls the wrapped function.

type PlacementMove added in v0.33.0 

type PlacementMove struct {
	Decision     cachemeta.PlacementDecision
	SpanDigest   string
	From         int
	N            int
	ModelID      string
	TokenizerID  string
	PositionMode cachemeta.PositionMode
	Owner        string
	Lease        string
}

PlacementMove bundles a cachemeta.PlacementDecision with the identity of the span it moves. The decision names the TIERS and the ACTION; the adapter needs the SPAN — its digest, its [From,From+N) range in the live cache, and the model/tokenizer/position mode it was derived under — to act on it. It is the field-only lowering shape a capacity-pressure loop builds per pressured entry before handing it to Execute.

type PlacementResult added in v0.33.0 

type PlacementResult struct {
	Recorded CacheEventResult
	Evicted  int
	Applied  bool
}

PlacementResult is the typed outcome of executing a PlacementMove. Recorded carries the normalized cache-entry and its lookup verdict — so a staging FAULT is observable (never a silent recompute). Evicted is the number of positions removed from the live KV cache. Applied reports whether the live span was actually moved/dropped: true for a completed demote/spill/evict; false when the move was not executed (promote/keep, or a demote/spill whose staging faulted and whose live copy was therefore retained).

type PrefixResidency added in v0.35.0 

type PrefixResidency struct {
	WorkerID    string
	Digest      string
	ModelID     string
	TokenizerID string
	Medium      string
	Tokens      int64
	BlockSize   int
	GroupIdx    int
	UpdatedAt   time.Time
}

PrefixResidency is one worker's current claim that a prefix/KV block is resident.

func RecordSGLangRadixSnapshot added in v0.35.0 

func RecordSGLangRadixSnapshot(worker, model string, idx *PrefixResidencyIndex, snap SGLangRadixSnapshot) []PrefixResidency

RecordSGLangRadixSnapshot is the pure fold of a radix snapshot into the per-worker residency index. The snapshot is a full residency picture, so the worker's prior rows are CLEARED before the snapshot rows are stored.

type PrefixResidencyIndex added in v0.35.0 

type PrefixResidencyIndex struct {
	// contains filtered or unexported fields
}

PrefixResidencyIndex is the per-worker prefix-residency index fed by vLLM KV-cache events.

func NewPrefixResidencyIndex added in v0.35.0 

func NewPrefixResidencyIndex() *PrefixResidencyIndex

func (*PrefixResidencyIndex) Clear added in v0.35.0 

func (idx *PrefixResidencyIndex) Clear(worker string)

func (*PrefixResidencyIndex) Has added in v0.35.0 

func (idx *PrefixResidencyIndex) Has(worker, digest string) bool

func (*PrefixResidencyIndex) Remove added in v0.35.0 

func (idx *PrefixResidencyIndex) Remove(worker string, digests ...string)

func (*PrefixResidencyIndex) Snapshot added in v0.35.0 

func (idx *PrefixResidencyIndex) Snapshot(worker string) []PrefixResidency

func (*PrefixResidencyIndex) Store added in v0.35.0 

func (idx *PrefixResidencyIndex) Store(worker string, rows ...PrefixResidency)

type SGLangCacheObserver added in v0.38.0 

type SGLangCacheObserver struct {
	// contains filtered or unexported fields
}

SGLangCacheObserver reads an SGLang upstream's radix/prefix-cache reuse signal from its public Prometheus surface and reports it as a wire-neutral engine.CacheCapability. It satisfies CacheCapabilityProducer behind the item-32 seam (#1550), so the gateway can report an SGLang-fronted session's cache capability without importing this package.

func NewSGLangCacheObserver added in v0.38.0 

func NewSGLangCacheObserver(cfg SGLangConfig) *SGLangCacheObserver

NewSGLangCacheObserver builds an observer over an SGLang worker's public /metrics surface.

func (*SGLangCacheObserver) CacheCapability added in v0.38.0 

func (o *SGLangCacheObserver) CacheCapability() CacheCapability

CacheCapability satisfies engine.CacheCapabilityProducer. Before Observe has resolved a signal it FAILS CLOSED to the unknown/unavailable label rather than inferring a positive — the same fail-closed default the contract's zero verdict has.

func (*SGLangCacheObserver) Observe added in v0.38.0 

func (o *SGLangCacheObserver) Observe(ctx context.Context) (SGLangRadixCacheSignal, error)

Observe scrapes the SGLang Prometheus endpoint once and caches the radix-cache signal for CacheCapability. A transport error is returned; a NON-200 status (metrics disabled) is NOT an error — it resolves to the unavailable/no-evidence signal, because a disabled endpoint is a legitimate "engine exposes no cache surface" observation rather than a failed read. SGLang's HTTP root is not an OpenAI /v1 frontend, so no /v1 suffix is stripped when the /metrics URL is derived (matching sglang.go's metricsURL). Safe to call again to refresh.

type SGLangConfig added in v0.35.0 

type SGLangConfig struct {
	BaseURL    string
	Model      string
	APIKey     string
	WorkerID   string
	MetricsURL string
	// RadixURL is the per-worker RadixAttention residency endpoint. SGLang exposes
	// no single standardized public radix-dump path, so this is supplied by the
	// router/bridge (or a test stub) and consumed as a decoded snapshot.
	RadixURL string
	Client   *http.Client

	CacheRecorder *CacheEventRecorder
	Residency     *PrefixResidencyIndex
}

SGLangConfig wires one SGLang worker through public surfaces only: the native /generate endpoint for generation, a radix-residency poll for RadixAttention's cache-aware signal, and Prometheus for serving metrics. It deliberately does not rely on an SGLang source patch or an in-process Python API.

func EnvSGLangConfig added in v0.35.0 

func EnvSGLangConfig() SGLangConfig

EnvSGLangConfig returns the default SGLang driver configuration. FAK_SGLANG_BASE_URL should point at the worker's HTTP root (the host:port sglang.launch_server binds).

type SGLangEngine added in v0.35.0 

type SGLangEngine struct {
	// contains filtered or unexported fields
}

SGLangEngine is the SGLang adapter behind abi.EngineDriver/LifecycleEngine.

func NewSGLangEngine added in v0.35.0 

func NewSGLangEngine(cfg SGLangConfig) *SGLangEngine

NewSGLangEngine builds an SGLang driver over public SGLang surfaces.

func (*SGLangEngine) Admit added in v0.35.0 

func (e *SGLangEngine) Admit(ctx context.Context, c *abi.ToolCall) (abi.EngineRequest, error)

Admit submits one request to SGLang's native /generate with stream=true and returns a live request handle whose Tokens channel receives SSE deltas as SGLang emits them.

func (*SGLangEngine) CacheRecorder added in v0.35.0 

func (e *SGLangEngine) CacheRecorder() *CacheEventRecorder

CacheRecorder exposes the cache-event recorder fed by RadixAttention prefix-cache hits surfaced on each /generate turn.

func (*SGLangEngine) Caps added in v0.35.0 

func (e *SGLangEngine) Caps() []abi.Capability

Caps advertises the SGLang adapter, the native /generate surface, lifecycle streaming, RadixAttention residency ingestion, metrics scrape normalization, and the honest whole-prefix cache-control boundary.

func (*SGLangEngine) Complete added in v0.35.0 

func (e *SGLangEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

Complete drains the live stream and returns the assembled result.

func (*SGLangEngine) PollRadixResidency added in v0.35.0 

func (e *SGLangEngine) PollRadixResidency(ctx context.Context) (SGLangRadixSnapshot, error)

PollRadixResidency fetches one radix-residency snapshot from the configured endpoint and folds it into the per-worker prefix-residency index. It is the RadixAttention cache-aware signal consumed for cache-aware routing.

func (*SGLangEngine) Residency added in v0.35.0 

func (e *SGLangEngine) Residency() *PrefixResidencyIndex

Residency exposes the per-worker prefix-residency index the radix poll feeds, so a router can read it for cache-aware dispatch.

func (*SGLangEngine) RunRadixPoll added in v0.35.0 

func (e *SGLangEngine) RunRadixPoll(ctx context.Context, interval time.Duration) error

RunRadixPoll polls the radix endpoint on an interval until ctx is cancelled, keeping the residency index current. A poll error (other than cancellation) ends the loop so the caller can re-establish the subscription.

func (*SGLangEngine) ScrapeServingMetrics added in v0.35.0 

func (e *SGLangEngine) ScrapeServingMetrics(ctx context.Context) (ServingMetricsSnapshot, error)

ScrapeServingMetrics reads SGLang's Prometheus endpoint and normalizes its scheduler metrics into fak's shared engine-serving schema (fak_serving_*).

func (*SGLangEngine) WeightBearing added in v0.37.0 

func (e *SGLangEngine) WeightBearing() bool

WeightBearing declares that SGLang runs a model-forward, not a deterministic tool.

type SGLangRadixCacheSignal added in v0.38.0 

type SGLangRadixCacheSignal struct {
	// Present is true when the scrape exposed sglang:cache_hit_rate or
	// sglang:prefix_cache_hit_rate. A present-but-zero gauge is still Present.
	Present bool
	// HitRatio is the observed radix/prefix cache-hit ratio in 0..1, normalized from a
	// percentage when the gauge reports 1..100 (the same normalization
	// ParseSGLangPrometheus applies). Meaningful only when Present; zero otherwise, and
	// never emitted as a reuse number then.
	HitRatio float64
	// Note explains a signal-ABSENT observation (metrics endpoint disabled, gauge not in
	// the build). Empty when Present.
	Note string
}

SGLangRadixCacheSignal is the decoded SGLang radix/prefix-cache observation: whether the upstream's Prometheus surface exposed the radix hit-rate gauge at all, and, when it did, the observed hit ratio. Present is the ONLY thing that separates "observed reuse" from "unavailable": the gauge MAY be present and zero (a live surface reporting no reuse yet), which is a passive-observe observation, not an absent signal.

func ObserveSGLangRadixCache added in v0.38.0 

func ObserveSGLangRadixCache(metricsText string) SGLangRadixCacheSignal

ObserveSGLangRadixCache scans an SGLang Prometheus scrape for the radix/prefix-cache hit-rate gauge and returns the decoded signal. It marks the signal Present the moment the gauge appears — even at zero — so a live-but-cold radix cache is distinguished from a build that never exposes the gauge. It reuses the same gauge names and 0..100→0..1 normalization as sglang.go's ParseSGLangPrometheus so the observation and the serving-metrics view never drift.

func (SGLangRadixCacheSignal) Capability added in v0.38.0 

func (s SGLangRadixCacheSignal) Capability() CacheCapability

Capability maps the decoded SGLang radix-cache signal onto the wire-neutral engine.CacheCapability contract. A present gauge is CachePassiveObserve (fak observes symptoms; SGLang's public control plane is whole-prefix flush_cache reset only, SupportsExactSpan=false — never an active class); an absent one is the fail-closed CacheUnknown with an explicit "unavailable" Evidence string. Both carry ProvenanceProvider (an observed provider gauge, not a kernel witness) and ColdPathCorrect true (observation never changes the request's cold path — a radix miss still sends full required context).

type SGLangRadixSnapshot added in v0.35.0 

type SGLangRadixSnapshot struct {
	WorkerID    string                 `json:"worker_id,omitempty"`
	ModelID     string                 `json:"model_id,omitempty"`
	TokenizerID string                 `json:"tokenizer_id,omitempty"`
	TS          float64                `json:"ts,omitempty"`
	Resident    []SGLangResidentPrefix `json:"resident"`
}

SGLangRadixSnapshot is the decoded per-worker RadixAttention residency picture fak consumes from a poll of the radix endpoint. It is a full snapshot (the resident prefix set for the worker), so folding it REPLACES that worker's residency rows.

type SGLangResidentPrefix added in v0.35.0 

type SGLangResidentPrefix struct {
	Digest    string `json:"digest,omitempty"`
	Hash      string `json:"hash,omitempty"`
	Tokens    int64  `json:"tokens,omitempty"`
	Medium    string `json:"medium,omitempty"`
	BlockSize int    `json:"block_size,omitempty"`
}

SGLangResidentPrefix is one resident radix prefix/block in a snapshot.

type ServingMetricsSnapshot added in v0.35.0 

type ServingMetricsSnapshot struct {
	Engine   string
	WorkerID string
	// WorkerRole is optional. It is set by P/D-aware control planes such as
	// Dynamo when the upstream exposes prefill/decode role labels.
	WorkerRole string

	TTFT  metricSumCount
	TPOT  metricSumCount
	ITL   metricSumCount
	Queue metricSumCount

	KVCacheUsage     float64
	RequestsRunning  float64
	RequestsWaiting  float64
	RequestsSwapped  float64
	RequestSuccesses float64
	PrefixQueries    float64
	PrefixHits       float64

	// PrefixCacheHitRatio is a directly-reported prefix/radix cache-hit ratio
	// (0..1) for engines that expose it as a single gauge (e.g. SGLang's
	// sglang:cache_hit_rate) instead of query/hit counters. It is a pointer so an
	// engine that reports counters instead (vLLM) leaves it nil and emits NO ratio
	// line — a literal 0.0 would read as a measured 0% hit rate, which it is not.
	PrefixCacheHitRatio *float64

	// Optional P/D worker-load gauges. Dynamo exposes these as per-worker signals:
	// active decode blocks and queued prefill tokens. They are not ratios, so they
	// stay separate from KVCacheUsage while still rendering in the fak_serving_* L2
	// namespace with worker labels.
	ActiveDecodeBlocks  *float64
	ActivePrefillTokens *float64
}

ServingMetricsSnapshot is the normalized serving L2 schema for ridden engines.

func ParseLLMDPrometheus added in v0.37.0 

func ParseLLMDPrometheus(workerID, text string) ServingMetricsSnapshot

ParseLLMDPrometheus maps vLLM-style worker metrics from an llm-d deployment into fak's normalized serving schema, preserving llm-d as the engine identity.

func ParseSGLangPrometheus added in v0.35.0 

func ParseSGLangPrometheus(workerID, text string) ServingMetricsSnapshot

ParseSGLangPrometheus extracts the SGLang scheduler metric names and maps them onto the SAME fak_serving_* schema the vLLM emitter targets (two emitters, one schema). Unmapped sglang:* names are ignored — only fields the shared schema already carries are populated, so the output never leaks SGLang-only metric names.

func ParseVLLMPrometheus added in v0.35.0 

func ParseVLLMPrometheus(workerID, text string) ServingMetricsSnapshot

ParseVLLMPrometheus extracts the vLLM metric names used by vLLM V1 and maps them onto a stable fak_serving_* schema.

func (ServingMetricsSnapshot) Prometheus added in v0.35.0 

func (s ServingMetricsSnapshot) Prometheus() string

Prometheus renders normalized metrics. The values are relabeled as fak_serving_* so a vLLM worker and future SGLang/native emitters can share one schema.

type ServingMetricsSnapshots added in v0.35.0 

type ServingMetricsSnapshots []ServingMetricsSnapshot

ServingMetricsSnapshots renders one or more worker rows in the normalized fak_serving_* schema without duplicating HELP/TYPE records.

func (ServingMetricsSnapshots) Prometheus added in v0.35.0 

func (rows ServingMetricsSnapshots) Prometheus() string

type UpstreamFactory added in v0.33.0 

type UpstreamFactory func(ctx context.Context, c *abi.ToolCall) (UpstreamStream, error)

UpstreamFactory opens an upstream stream for one admitted call. A real adapter submits the prompt to the upstream engine and returns a reader over its async token stream. An error here fails Admit (the request was never admitted).

type UpstreamStream added in v0.33.0 

type UpstreamStream interface {
	// Next returns the next decoded step. A non-nil err (including ctx.Err() on
	// abort) terminates the request; otherwise UpstreamToken.Done signals the
	// upstream finished the turn.
	Next(ctx context.Context) (UpstreamToken, error)
}

UpstreamStream is the minimal async-token surface an external serving engine exposes for one request: pull the next decoded step, or learn the stream is finished, honoring ctx for mid-stream abort. A real adapter implements Next by reading the upstream's token stream; cancelling ctx aborts the upstream request.

type UpstreamToken added in v0.33.0 

type UpstreamToken struct {
	ID   int    // the decoded token id
	Text string // optional incremental detok text ("" if none)
	Done bool   // the upstream finished the turn (no token in this event)
}

UpstreamToken is one decoded step from an upstream stream. It is a STRUCT, not a positional return, EXACTLY so an adapter author can grow it with upstream-native fields (per-token logprobs/top-k, a finish-reason, usage) WITHOUT breaking every UpstreamStream implementation — a return-tuple could not grow additively. Done marks the turn finished and carries no token.

type Usage 

type Usage struct {
	InputTokens         int `json:"prompt_tokens"`
	OutputTokens        int `json:"completion_tokens"`
	TotalTokens         int `json:"total_tokens"`
	CacheReadTokens     int `json:"cache_read_tokens,omitempty"`
	CacheCreationTokens int `json:"cache_creation_tokens,omitempty"`
}

Usage is the token accounting extracted from a completion (unit 42). CacheReadTokens / CacheCreationTokens carry the provider prompt-prefix cache axes (issue #1846) so a cassette built from a captured live session can report the SAME 4-way usage split (input / cache_read / cache_creation / output) the provider billed, not just the mock engine's synthesized input/output pair.

type VLLMCacheObserver added in v0.38.0 

type VLLMCacheObserver struct {
	// contains filtered or unexported fields
}

VLLMCacheObserver reads a vLLM upstream's prefix-cache reuse signal from its public Prometheus surface and reports it as a wire-neutral engine.CacheCapability. It satisfies CacheCapabilityProducer behind the item-32 seam (#1550), so the gateway can report a vLLM-fronted session's cache capability without importing this package.

func NewVLLMCacheObserver added in v0.38.0 

func NewVLLMCacheObserver(cfg VLLMConfig) *VLLMCacheObserver

NewVLLMCacheObserver builds an observer over a vLLM worker's public /metrics surface.

func (*VLLMCacheObserver) CacheCapability added in v0.38.0 

func (o *VLLMCacheObserver) CacheCapability() CacheCapability

CacheCapability satisfies engine.CacheCapabilityProducer. Before Observe has resolved a signal it FAILS CLOSED to the unknown/unavailable label rather than inferring a positive — the same fail-closed default the contract's zero verdict has.

func (*VLLMCacheObserver) Observe added in v0.38.0 

func (o *VLLMCacheObserver) Observe(ctx context.Context) (VLLMPrefixCacheSignal, error)

Observe scrapes the vLLM Prometheus endpoint once and caches the prefix-cache signal for CacheCapability. A transport error is returned; a NON-200 status (metrics disabled) is NOT an error — it resolves to the unavailable/no-evidence signal, because a disabled endpoint is a legitimate "engine exposes no cache surface" observation rather than a failed read. Safe to call again to refresh.

type VLLMConfig added in v0.35.0 

type VLLMConfig struct {
	BaseURL    string
	Model      string
	APIKey     string
	WorkerID   string
	MetricsURL string
	Client     *http.Client

	// PriorityScheduling advertises that the served vLLM engine runs the V1
	// priority scheduler, so a fak TurnIntent priority may be lowered to the
	// request. When false the adapter emits no priority field (FCFS default).
	PriorityScheduling bool

	CacheRecorder *CacheEventRecorder
	Residency     *PrefixResidencyIndex
	KVEvents      VLLMKVEventSource
}

VLLMConfig wires one vLLM V1 worker through public surfaces only: OpenAI-compatible HTTP for generation, Prometheus for serving metrics, and KV-cache event batches for residency. It deliberately does not rely on vLLM source patches or internal Python APIs.

Honesty boundary: current vLLM exposes whole-prefix cache reset through its public control plane. Exact-span KV governance is not asserted here; it remains enginecache.SupportsExactSpan=false and degrades to whole-prefix reset.

func EnvVLLMConfig added in v0.35.0 

func EnvVLLMConfig() VLLMConfig

EnvVLLMConfig returns the default vLLM driver configuration. FAK_VLLM_BASE_URL should point at the worker's OpenAI-compatible root, usually http://host:port/v1.

type VLLMDeterminism added in v0.37.0 

type VLLMDeterminism string

VLLMDeterminism is the vLLM served-engine determinism mode fak surfaces as a capability (issue #1734). It is an OPERATOR-declared / engine-reported property of how the worker was LAUNCHED, NOT something fak can infer from a request's sampling parameters. vLLM documents two distinct reproducibility guarantees on its public surfaces:

  • batch invariance for online reproducibility (docs/usage/reproducibility.md),
  • a deterministic offline scheduling option,

and, critically, that WITHOUT one of those, batching variation and numerical instability can change outputs even at temperature 0 (docs/usage/faq.md). fak therefore treats "unavailable" as the honest default: absent a positive signal that the engine is running batch-invariant or deterministic-offline, a temperature-0 request is NOT reproducible token-for-token.

This is a correctness/operability bridge, not a kernel rebuild: fak uses the engine capability when the operator declares it and labels it "unavailable" when they do not -- it does not reimplement batch-invariant kernels in fak. 

const (
	// DeterminismUnavailable is the honest default: the engine reports (or the
	// operator has declared) no batch-invariance and no deterministic scheduler,
	// so outputs may vary across batches even at temperature 0.
	DeterminismUnavailable VLLMDeterminism = "unavailable"

	// DeterminismBatchInvariant is vLLM's batch-invariant kernel mode for online
	// reproducibility: identical requests reproduce regardless of co-batched load.
	DeterminismBatchInvariant VLLMDeterminism = "batch_invariance"

	// DeterminismDeterministicOffline is vLLM's deterministic offline scheduling
	// option: a fixed schedule makes an offline run reproducible.
	DeterminismDeterministicOffline VLLMDeterminism = "deterministic_offline_scheduler"
)

func EnvVLLMDeterminism added in v0.37.0 

func EnvVLLMDeterminism() VLLMDeterminism

EnvVLLMDeterminism reads FAK_VLLM_DETERMINISM and normalizes it. Unset or unrecognized => DeterminismUnavailable (the fail-closed default). vLLM's batch-invariance / deterministic-scheduler posture is a server LAUNCH property, so fak reads it from the deployment environment rather than a per-request field.

func ParseVLLMDeterminism added in v0.37.0 

func ParseVLLMDeterminism(s string) VLLMDeterminism

ParseVLLMDeterminism normalizes a free-text determinism label (env value, config string, or artifact field) onto one of the three canonical modes. Anything it does not recognize -- including the empty string -- maps to DeterminismUnavailable, so an unknown or missing declaration fails CLOSED to "not reproducible" rather than silently asserting determinism.

func (VLLMDeterminism) Capability added in v0.37.0 

func (d VLLMDeterminism) Capability() abi.Capability

Capability returns the negotiable capability token for this determinism mode, e.g. "engine.vllm.determinism.batch_invariance". Every mode -- including "unavailable" -- advertises a token so a consumer always learns the engine's declared reproducibility posture rather than having to infer its absence.

func (VLLMDeterminism) GuaranteesReproducibleOutput added in v0.37.0 

func (d VLLMDeterminism) GuaranteesReproducibleOutput() bool

GuaranteesReproducibleOutput reports whether this mode makes identical requests reproduce token-for-token. True only for batch-invariance or the deterministic offline scheduler; false for "unavailable".

func (VLLMDeterminism) Normalize added in v0.37.0 

func (d VLLMDeterminism) Normalize() VLLMDeterminism

Normalize returns the mode itself, treating the empty zero value (and any unrecognized value) as the fail-closed DeterminismUnavailable so a hand-built VLLMDeterminism still reports an honest capability.

func (VLLMDeterminism) TemperatureZeroYieldsDeterminism added in v0.37.0 

func (d VLLMDeterminism) TemperatureZeroYieldsDeterminism() bool

TemperatureZeroYieldsDeterminism is the witness guard #1734 requires: it reports whether, under this determinism mode, a temperature-0 request actually reproduces token-for-token. It is FALSE for DeterminismUnavailable -- temperature 0 alone, when the engine reports dynamic batching without batch invariance, does NOT make outputs reproducible. A replay/witness claim MUST consult this before citing temperature 0 as determinism; a false result means the claim is not witnessed.

type VLLMEngine added in v0.35.0 

type VLLMEngine struct {
	// contains filtered or unexported fields
}

VLLMEngine is a vLLM V1 adapter behind abi.EngineDriver/LifecycleEngine.

func NewVLLMEngine added in v0.35.0 

func NewVLLMEngine(cfg VLLMConfig) *VLLMEngine

NewVLLMEngine builds a vLLM driver over public vLLM surfaces.

func (*VLLMEngine) Admit added in v0.35.0 

func (e *VLLMEngine) Admit(ctx context.Context, c *abi.ToolCall) (abi.EngineRequest, error)

Admit submits one request to vLLM with stream=true and returns a live request handle whose Tokens channel receives SSE deltas as vLLM emits them.

func (*VLLMEngine) Caps added in v0.35.0 

func (e *VLLMEngine) Caps() []abi.Capability

Caps advertises the vLLM adapter, the OpenAI HTTP surface, lifecycle streaming, KV-event ingestion, metrics scrape normalization, and the honest whole-prefix cache-control boundary.

func (*VLLMEngine) Complete added in v0.35.0 

func (e *VLLMEngine) Complete(ctx context.Context, c *abi.ToolCall) (*abi.Result, error)

Complete drains the live stream and returns the assembled result.

func (*VLLMEngine) Determinism added in v0.37.0 

func (e *VLLMEngine) Determinism() VLLMDeterminism

Determinism reports the determinism mode configured for this vLLM worker via the environment (FAK_VLLM_DETERMINISM), normalized to the fail-closed default when unset. This is the EngineDriver-level determinism accessor.

func (*VLLMEngine) DeterminismCapability added in v0.37.0 

func (e *VLLMEngine) DeterminismCapability() abi.Capability

DeterminismCapability reports this vLLM engine's determinism posture as a single negotiable capability token (criterion 1 of #1734): one of engine.vllm.determinism.{batch_invariance,deterministic_offline_scheduler,unavailable}. It is additive to Caps(): a consumer that already reads Caps() can intersect it against the three registered determinism tokens, and a consumer that wants only the determinism axis calls this directly without scanning the whole capability set.

func (*VLLMEngine) RecordVLLMKVEventBatch added in v0.35.0 

func (e *VLLMEngine) RecordVLLMKVEventBatch(batch VLLMKVEventBatch) []CacheEventResult

RecordVLLMKVEventBatch folds one decoded vLLM KV-event batch into the per-worker residency index and the shared cache-event recorder.

func (*VLLMEngine) RunKVEventSubscription added in v0.35.0 

func (e *VLLMEngine) RunKVEventSubscription(ctx context.Context) error

RunKVEventSubscription consumes decoded vLLM KV-event batches until ctx is cancelled or the source ends. The native vLLM transport is ZMQ/msgpack; fak stays dependency-free by taking the decoded batch stream at this seam, so a process-local bridge or test fixture can feed the same residency/index logic.

func (*VLLMEngine) ScrapeServingMetrics added in v0.35.0 

func (e *VLLMEngine) ScrapeServingMetrics(ctx context.Context) (ServingMetricsSnapshot, error)

ScrapeServingMetrics reads vLLM's Prometheus endpoint and normalizes the TTFT/TPOT/ITL/queue/KV-util counters into fak's engine-serving schema.

func (*VLLMEngine) WeightBearing added in v0.37.0 

func (e *VLLMEngine) WeightBearing() bool

WeightBearing declares that vLLM runs a model-forward, not a deterministic tool.

type VLLMJSONKVEventSource added in v0.35.0 

type VLLMJSONKVEventSource struct {
	// contains filtered or unexported fields
}

VLLMJSONKVEventSource reads one JSON-encoded VLLMKVEventBatch per line. It is the dependency-free bridge/test transport for a ZMQ/msgpack subscriber that has already decoded vLLM's native EventBatch payloads.

func NewVLLMJSONKVEventSource added in v0.35.0 

func NewVLLMJSONKVEventSource(r io.ReadCloser) *VLLMJSONKVEventSource

NewVLLMJSONKVEventSource wraps an NDJSON batch stream as a KV event source.

func (*VLLMJSONKVEventSource) Close added in v0.35.0 

func (s *VLLMJSONKVEventSource) Close() error

func (*VLLMJSONKVEventSource) Next added in v0.35.0 

func (s *VLLMJSONKVEventSource) Next(ctx context.Context) (VLLMKVEventBatch, error)

type VLLMKVEvent added in v0.35.0 

type VLLMKVEvent struct {
	Type                         string `json:"type,omitempty"`
	Event                        string `json:"event,omitempty"`
	Kind                         string `json:"kind,omitempty"`
	BlockHashes                  []any  `json:"block_hashes,omitempty"`
	ParentBlockHash              any    `json:"parent_block_hash,omitempty"`
	TokenIDs                     []int  `json:"token_ids,omitempty"`
	BlockSize                    int    `json:"block_size,omitempty"`
	LoraID                       *int   `json:"lora_id,omitempty"`
	Medium                       string `json:"medium,omitempty"`
	LoraName                     string `json:"lora_name,omitempty"`
	GroupIdx                     *int   `json:"group_idx,omitempty"`
	KVCacheSpecKind              string `json:"kv_cache_spec_kind,omitempty"`
	KVCacheSpecSlidingWindow     *int   `json:"kv_cache_spec_sliding_window,omitempty"`
	KVCacheSpecSlidingWindowJSON *int   `json:"kv_cache_spec_sliding_window_json,omitempty"`
}

VLLMKVEvent mirrors vLLM's BlockStored, BlockRemoved, and AllBlocksCleared event variants. Block hashes are kept as JSON values because vLLM's ExternalBlockHash is versioned; fak only needs a stable digest string.

type VLLMKVEventBatch added in v0.35.0 

type VLLMKVEventBatch struct {
	TS               float64        `json:"ts"`
	Events           []VLLMKVEvent  `json:"events"`
	DataParallelRank *int           `json:"data_parallel_rank,omitempty"`
	WorkerID         string         `json:"worker_id,omitempty"`
	ModelID          string         `json:"model_id,omitempty"`
	TokenizerID      string         `json:"tokenizer_id,omitempty"`
	Raw              map[string]any `json:"-"`
}

VLLMKVEventBatch is the JSON-shaped mirror of vLLM's KVEventBatch. vLLM publishes the live stream over ZMQ/msgpack; this type is the in-process normalized form the adapter consumes after transport decoding.

type VLLMKVEventSource added in v0.35.0 

type VLLMKVEventSource interface {
	Next(ctx context.Context) (VLLMKVEventBatch, error)
	Close() error
}

VLLMKVEventSource is a decoded vLLM KV-event stream. A stdlib-only fak build cannot import pyzmq/msgspec, so the transport decoder lives outside this leaf and hands the adapter EventBatch-shaped values here.

type VLLMPrefixCacheSignal added in v0.38.0 

type VLLMPrefixCacheSignal struct {
	// Present is true when the scrape exposed vllm:prefix_cache_queries or
	// vllm:prefix_cache_hits. A present-but-zero counter is still Present.
	Present bool
	// Queries and Hits are the observed prefix-cache counter totals, summed across the
	// scrape's label sets (a multi-rank worker emits one line per data-parallel rank).
	// Meaningful only when Present; both zero otherwise, and never emitted as a reuse
	// number then.
	Queries float64
	Hits    float64
	// Note explains a signal-ABSENT observation (metrics endpoint disabled, counters not
	// in the build). Empty when Present.
	Note string
}

VLLMPrefixCacheSignal is the decoded vLLM prefix-cache observation: whether the upstream's Prometheus surface exposed the prefix-cache counters at all, and, when it did, the observed query/hit totals. Present is the ONLY thing that separates "observed reuse" from "unavailable": the counters MAY be present and zero (a live surface reporting no reuse yet), which is a passive-observe observation, not an absent signal.

func ObserveVLLMPrefixCache added in v0.38.0 

func ObserveVLLMPrefixCache(metricsText string) VLLMPrefixCacheSignal

ObserveVLLMPrefixCache scans a vLLM Prometheus scrape for the prefix-reuse counters and returns the decoded signal. It marks the signal Present the moment either counter appears — even at zero — so a live-but-cold cache is distinguished from a build that never exposes the counters. It reuses the same parse helper and counter names as vllm.go's ParseVLLMPrometheus so the observation and the serving-metrics view never drift.

func (VLLMPrefixCacheSignal) Capability added in v0.38.0 

func (s VLLMPrefixCacheSignal) Capability() CacheCapability

Capability maps the decoded vLLM prefix-cache signal onto the wire-neutral engine.CacheCapability contract. A present counter surface is CachePassiveObserve (fak observes symptoms; vLLM's public control plane is whole-prefix reset only, SupportsExactSpan=false — never an active class); an absent one is the fail-closed CacheUnknown with an explicit "unavailable" Evidence string. Both carry ProvenanceProvider (an observed provider counter, not a kernel witness) and ColdPathCorrect true (observation never changes the request's cold path — a miss still sends full required context).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.