residency

Documentation

Overview

Package residency contains modeld's backend-neutral KV residency policy.

It deliberately owns only logical decisions: which token ranges should remain hot under a derived budget, and which ranges may be moved cold. Backend adapters execute those decisions only when their engine exposes the necessary KV controls.

Index

• func ClassForSegment(kind string, stable bool, explicit string) contextasm.CacheClass
• func Drive(ctx context.Context, exec Executor, plan Plan) error
• func ParseCacheClass(tag string) (contextasm.CacheClass, bool)
• type AttentionScorer
• type Block
  • func BlocksFromManifest(m contextasm.ContextManifest, opts ManifestOptions) ([]Block, error)
• type BlockFlags
  • func (f BlockFlags) Has(want BlockFlags) bool
• type Capabilities
• type Controller
• type EvictionBudget
  • func DeriveEvictionBudget(windowTokens, slidingWindowTokens, blockSize int) EvictionBudget
  • func (b EvictionBudget) Valid() bool
• type Executor
• type ManifestOptions
• type MissingTokenRangesError
  • func (e *MissingTokenRangesError) Error() string
• type Plan
  • func PlanHotSet(in PlanInput) (Plan, error)
• type PlanInput
• type Range
  • func EvictColdRanges(plan Plan) []Range
  • func (r Range) Len() int
• type StreamPolicy

Functions

func ClassForSegment

func ClassForSegment(kind string, stable bool, explicit string) contextasm.CacheClass

ClassForSegment returns the explicit manifest cache class when valid, else a conservative default derived from segment kind and stable/volatile placement.

func Drive

func Drive(ctx context.Context, exec Executor, plan Plan) error

Drive executes the cold-eviction half of a residency plan against a self-locking Executor: it evicts each EvictCold range to the cold store, freeing hot KV. It is a no-op when the plan evicts nothing. Use it only from callers that do NOT already hold the session lock, since Executor methods lock internally.

func ParseCacheClass

func ParseCacheClass(tag string) (contextasm.CacheClass, bool)

ParseCacheClass parses the stable manifest cache-class tag.

Types

type AttentionScorer

type AttentionScorer interface {
	BlockAttentionScores(ctx context.Context, ranges []Range) ([]float32, error)
}

AttentionScorer is an optional backend seam for attention-aware eviction. Backends that cannot expose scores leave Capabilities.AttentionScores false and the planner falls back to recency/class ordering.

type Block

type Block struct {
	Range        Range
	Kind         string
	Stable       bool
	CacheClass   contextasm.CacheClass
	TokenHash    string
	LastUsed     int64
	Flags        BlockFlags
	Segment      int
	SplitOrdinal int
}

Block is the planner's unit of residency. Ranges must not overlap.

func BlocksFromManifest

func BlocksFromManifest(m contextasm.ContextManifest, opts ManifestOptions) ([]Block, error)

BlocksFromManifest converts backend-tokenized manifest segments into logical residency blocks. Missing CacheClass tags are normalized from kind/stability.

type BlockFlags

type BlockFlags uint16

BlockFlags are policy hints applied above CacheClass. Sinks and recent-window blocks are protected because sparse/streaming attention requires them hot.

const (
	FlagPinned BlockFlags = 1 << iota
	FlagSink
	FlagRecent
	FlagRetrieved
)

func (BlockFlags) Has

func (f BlockFlags) Has(want BlockFlags) bool

type Capabilities

type Capabilities struct {
	RemoveTail                   bool
	RemoveMiddle                 bool
	PositionShift                bool
	SparseAttention              bool
	SlidingWindowAttentionTokens int
	ColdStore                    bool
	RecomputeRange               bool
	AttentionScores              bool
}

Capabilities describes what a backend adapter can actually execute.

type Controller

type Controller interface {
	Capabilities() Capabilities
}

Controller is the optional engine-facing seam. It is intentionally not part of runtime/transport.Session.

type EvictionBudget

type EvictionBudget struct {
	SinkTokens   int // always-hot leading tokens (attention sinks)
	RecentTokens int // always-hot trailing window
	MaxTokens    int // hot budget; eviction keeps physical KV within this
}

EvictionBudget is the sink/recent/max split a backend uses to bound its hot KV while letting generation continue past the physical window. Both adapters derive it the same way so llama (imperative slide) and OpenVINO (declarative CacheEvictionConfig) enforce one policy.

func DeriveEvictionBudget

func DeriveEvictionBudget(windowTokens, slidingWindowTokens, blockSize int) EvictionBudget

DeriveEvictionBudget splits a served window into attention sinks, a recent window, and the total hot budget. It is eviction-algorithm policy (à la StreamingLLM/H2O), not hardware sizing: ~1/16 of the effective eviction window as sinks, ~1/4 as the recent window, Max = that window. Sliding-window models cap the eviction window at their model-native attention span because older windowed-layer KV cannot be useful hot context. blockSize aligns sizes for block-based caches (OpenVINO); pass <=1 for token-granular backends (llama). Windows too small to split keep everything hot (Valid() is false -> no eviction).

func (EvictionBudget) Valid

func (b EvictionBudget) Valid() bool

Valid reports whether the split can drive an eviction config: non-zero sizes with an evictable middle (Max > Sink + Recent).

type Executor

type Executor interface {
	Controller
	EvictRange(ctx context.Context, r Range) error
	AdmitRange(ctx context.Context, r Range) error
}

Executor is implemented only by adapters that can mutate physical KV ranges.

type ManifestOptions

type ManifestOptions struct {
	// ResidentTokens limits the manifest to the token range currently resident.
	// This lets callers run the planner after EnsurePrefix before volatile
	// segments have token ranges.
	ResidentTokens int
	// BlockSize splits large manifest segments into uniform logical blocks.
	// A non-positive value keeps each segment as one block.
	BlockSize int
	// LastUsed is copied onto every generated block. Callers that track richer
	// access recency can rewrite LastUsed before planning.
	LastUsed int64
	// RequireComplete reports missing ranges for every non-empty segment. Leave
	// false for prefix-only planning after EnsurePrefix, where volatile suffix
	// ranges have not been tokenized yet.
	RequireComplete bool
}

ManifestOptions controls block construction from a transport manifest.

type MissingTokenRangesError

type MissingTokenRangesError struct {
	Segments []string
}

MissingTokenRangesError reports non-empty manifest segments that are within the resident region but have not yet been backend-tokenized.

func (*MissingTokenRangesError) Error

func (e *MissingTokenRangesError) Error() string

type Plan

type Plan struct {
	BudgetTokens    int
	TotalTokens     int
	HotTokens       int
	ProtectedTokens int
	OverBudget      bool
	KeepHot         []Block
	EvictCold       []Block
	Diagnostics     []string
}

Plan is the planner output: KeepHot plus EvictCold partitions the input blocks. HotTokens can exceed BudgetTokens only when protected blocks alone do.

func PlanHotSet

func PlanHotSet(in PlanInput) (Plan, error)

PlanHotSet produces the hot/cold partition for a token budget.

type PlanInput

type PlanInput struct {
	Blocks       []Block
	BudgetTokens int
	// SinkTokens and RecentTokens mark blocks overlapping those token spans as
	// protected. The spans are coarse by design; block splitting controls
	// precision.
	SinkTokens   int
	RecentTokens int
	StreamPolicy StreamPolicy
	Capabilities Capabilities
	// AttentionScores optionally supplies one score per input block. Higher
	// means more important. When present, eviction within each CacheClass drops
	// lower-score blocks first; when absent, ordering remains recency-based.
	AttentionScores []float32
}

PlanInput is the pure policy input. BudgetTokens must be derived by capacity planning, not chosen here.

type Range

type Range struct {
	Start int
	End   int
}

Range is a half-open token range [Start, End).

func EvictColdRanges

func EvictColdRanges(plan Plan) []Range

EvictColdRanges returns the ranges of a plan's EvictCold blocks ordered tail-first (highest Start first). A caller that evicts them in this order does not shift the indices of ranges it has not evicted yet: removing a higher range never moves the tokens below it. Empty or degenerate ranges are dropped.

Backend adapters driving eviction from inside an already-locked prefill path iterate this over their own *Locked eviction primitive; callers that do not hold the session lock use Drive.

func (Range) Len

func (r Range) Len() int

type StreamPolicy

type StreamPolicy struct {
	Enabled      bool
	SinkTokens   int
	RecentTokens int
}