Documentation
¶
Overview ¶
Package orchestrate decides which local models stay resident in limited device memory. It is the control-plane policy over the serve manager: given the models that should be resident and the models currently served, it computes the launches and evictions that converge the two within a memory budget, without ever evicting a pinned or actively decoding model. The decision is a pure function, so the policy is exhaustively testable without a live runtime; the reconcile-driven controller that applies it is wired separately.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Controller ¶
type Controller struct {
// contains filtered or unexported fields
}
Controller drives the resident set toward the desired state on a reconcile loop. Each pass re-reads the desired state and the observed resident set, computes a plan with Schedule, and applies it: evictions first to free memory, then launches. An apply failure is reported as a transient error so the loop backs off and retries, which is what makes the controller self-healing. Because Schedule is a fixed point, a converged set produces an empty plan, so a steady state does no work and cannot thrash.
func NewController ¶
NewController builds a controller that reconciles server toward what provider wants, scheduling its retries and resync on clk.
func (*Controller) Run ¶
func (c *Controller) Run(ctx context.Context)
Run drives the loop until ctx is cancelled. It triggers an initial reconcile so the current desired state is applied at once, then blocks until every worker has drained.
func (*Controller) Trigger ¶
func (c *Controller) Trigger()
Trigger asks for a reconcile now, for example after the desired state changes. Triggers collapse, so a burst of changes causes a single reconcile.
type Desired ¶
type Desired struct {
// ModelID is the catalog id, the identity shared with the serve manager.
ModelID string
// Footprint is the estimated device memory the model occupies when resident, in bytes.
// It comes from the model's known size, not a live reading, so a model can be budgeted
// before it is launched. A negative value is treated as zero.
Footprint int64
// Priority orders models under memory pressure: a higher priority is kept, and a lower
// one is evicted first.
Priority int
// Pinned keeps a model resident regardless of priority or budget, for a model that must
// stay hot (a small default model, or a draft model for speculative decoding).
Pinned bool
// Draft, when set, is a small companion model paired with this one for speculative
// decoding. It must be resident whenever its primary is, so the policy keeps and budgets
// the two together: the primary is admitted only if both fit, and evicting the primary
// evicts the draft. A nil Draft means the model serves on its own.
Draft *Draft
}
Desired is a model the controller should keep resident, with the inputs the policy needs to choose under memory pressure.
type DesiredState ¶
DesiredState is what the controller should make true: the models that should be resident and the device-memory budget they share. It is read fresh on every reconcile, so a change in what the system wants is picked up without an event.
type Draft ¶
type Draft struct {
// ModelID is the draft model's catalog id.
ModelID string
// Footprint is the device memory it occupies when resident, in bytes. A negative value is
// treated as zero.
Footprint int64
}
Draft is the companion model paired with a primary for speculative decoding. It is identified and budgeted on its own, but it is never scheduled independently: it rides with its primary.
type FailureKind ¶
type FailureKind int
FailureKind classifies why a model server failed, because the right reaction differs by cause: retrying an out-of-memory launch unchanged is futile, while a one-off crash is worth another try.
const ( // FailureNone means the model is not in a failed state. FailureNone FailureKind = iota // FailureCrash means the server process exited or failed to start, with no sign that // memory was the cause: a transient fault worth retrying a bounded number of times. FailureCrash // FailureOOM means the launch ran out of device or host memory. Retrying the same // footprint cannot succeed, so recovery must shrink it or fall back. FailureOOM // FailureHang means the server started but never became healthy: it may be a slow load // worth one retry, or a wedged runtime that needs a lighter config or a fallback. FailureHang )
type FootprintFunc ¶
FootprintFunc returns the device memory a model occupies when resident, in bytes. It is injected from the model catalog, the source of a model's known size.
type LaunchFunc ¶
type LaunchFunc func(ctx context.Context, modelID string, level LaunchLevel) error
LaunchFunc starts a model by id, resolving and serving it. It is injected because launching a catalog model (resolve, provision, build a plan, serve) is wired above this package; the orchestrator only needs the verb. The level asks for a smaller footprint as recovery escalates.
type LaunchLevel ¶
type LaunchLevel int
LaunchLevel is how aggressively a launch shrinks a model's footprint to make it fit. It climbs as recovery escalates, so a model that keeps running out of memory is retried smaller before it is given up on.
const ( // LaunchFull serves the model at its normal footprint. LaunchFull LaunchLevel = iota // LaunchDegraded serves with a reduced footprint (a smaller context window), the first // response to a memory or wedged-load failure. LaunchDegraded // LaunchMinimal serves at the smallest viable footprint, including running on the CPU, the // last resort for a model that will not fit in device memory. LaunchMinimal )
type Option ¶
type Option func(*Controller)
Option configures a Controller.
func WithClassifier ¶
func WithClassifier(f func(error) FailureKind) Option
WithClassifier sets how a launch error is mapped to a failure kind, which drives the recovery response. Without one, every launch error is treated as a crash (retried a bounded number of times, then quarantined); the serve layer injects a classifier that recognizes out-of-memory and wedged-load failures so they degrade instead of repeating.
func WithResync ¶
WithResync sets how often the controller re-reconciles without a trigger, the safety net that re-checks desired against observed. A value <= 0 disables periodic resync, leaving only the initial reconcile and explicit Trigger calls.
type Plan ¶
type Plan struct {
// Launch lists model ids to start; every entry is a desired model not already resident.
Launch []string
// Evict lists model ids to stop; every entry is currently resident and neither pinned
// nor active.
Evict []string
// Unschedulable lists desired models that did not fit the budget, so the caller can
// surface them rather than dropping them silently.
Unschedulable []string
}
Plan is the set of actions that converge the resident set toward the desired set. Applying it is idempotent: a resident set already equal to the chosen set yields an empty plan.
func Schedule ¶
Schedule computes the actions to converge resident toward desired within budget bytes of device memory. It first keeps every forced model (a pinned desired model, which overrides the budget, and every pinned or actively decoding resident model), then admits the remaining desired models in priority order (preferring those already resident, and the more-recently-used among ties, to avoid churn) for as long as they fit the budget. A resident model that is neither kept nor pinned nor active is evicted; a desired model that does not fit is reported as unschedulable. A model paired with a draft is admitted only if both fit, and a kept model's draft is kept with it, so a speculative-decoding pair is resident together or not at all. The result is deterministic and idempotent: the chosen set is a fixed point, so applying the plan and scheduling again yields no further launches or evictions.
type Provider ¶
type Provider interface {
Desired(ctx context.Context) (DesiredState, error)
}
Provider supplies the current desired state, read once per reconcile.
type RecoveryAction ¶
type RecoveryAction int
RecoveryAction is the next step to take for a failing model, in increasing severity. The controller maps each onto a concrete move: retry the same launch, relaunch with a reduced footprint, serve a smaller fallback model instead, or stop trying for now.
const ( // RecoverRetry tries the same launch again; the failure looked transient. RecoverRetry RecoveryAction = iota // RecoverDegrade relaunches with a smaller footprint (shorter context, lower memory // use), the first answer to memory pressure or a wedged load. RecoverDegrade // RecoverFallback gives up on this model for now and serves a smaller one in its place, // so something usable stays available rather than nothing. RecoverFallback // RecoverQuarantine stops launching the model until its state is reset, so a model that // fails no matter what does not consume the loop in a hot restart cycle. RecoverQuarantine )
func Recover ¶
func Recover(s RecoveryState) RecoveryAction
Recover decides the next action for a failing model, applying a graded ladder that escalates with repeated failure and never retries a cause that an identical retry cannot fix. An out-of-memory failure degrades, then falls back, then quarantines, and is never retried unchanged. A hang gets one retry (a slow load) before degrading and falling back. A plain crash is retried a bounded number of times, then quarantined. The action severity never decreases as attempts grow, so escalation is monotonic and a persistently failing model always reaches quarantine rather than looping forever.
type RecoveryState ¶
type RecoveryState struct {
Kind FailureKind
Attempts int
}
RecoveryState is a model's recent failure memory: the kind of the most recent failure and how many consecutive attempts have failed (including the one that produced this state). A successful launch resets it to the zero value.
type Resident ¶
type Resident struct {
// ModelID is the catalog id.
ModelID string
// Footprint is the device memory it occupies, in bytes. A negative value is treated as
// zero.
Footprint int64
// Pinned marks a model that must not be evicted.
Pinned bool
// Active is true when the model is currently decoding a request, so evicting it would
// drop in-flight work; an active model is kept even when it is no longer desired.
Active bool
// LastUsed orders eviction among otherwise-equal candidates, least-recently-used first.
// It is a logical or wall-clock stamp supplied by the caller.
LastUsed int64
}
Resident is a model the serve manager currently runs.
type ServeAdapter ¶
type ServeAdapter struct {
// contains filtered or unexported fields
}
ServeAdapter implements Server over a serve.Manager: it observes the resident set from the manager's records and live load stats, launches through an injected launcher, and evicts by stopping the manager's server. A model is reported active when it is currently decoding, so the scheduler never evicts in-flight work.
func NewServeAdapter ¶
func NewServeAdapter(mgr *serve.Manager, launch LaunchFunc, footprint FootprintFunc) *ServeAdapter
NewServeAdapter builds an adapter over mgr. launch starts a model by id and footprint reports its resident size; both come from the layer that owns the catalog and the serve plan. A nil launch makes Launch a no-op, and a nil footprint reads every model as zero-size.
func (*ServeAdapter) Evict ¶
func (a *ServeAdapter) Evict(_ context.Context, modelID string) error
Evict stops the model's server. Stopping a model that is not running is not an error, so a redundant evict is a safe no-op.
func (*ServeAdapter) Launch ¶
func (a *ServeAdapter) Launch(ctx context.Context, modelID string, level LaunchLevel) error
Launch starts a model by id through the injected launcher, passing the launch level down so a recovery relaunch can ask for a smaller footprint.
func (*ServeAdapter) Resident ¶
func (a *ServeAdapter) Resident(ctx context.Context) ([]Resident, error)
Resident lists the currently serving models with the inputs the scheduler needs. The memory footprint comes from the injected catalog lookup; whether a model is actively decoding comes from its live load stats, so an unreadable runtime is treated as idle (the scheduler may evict it) rather than as busy, which is the safe default for freeing memory.
type Server ¶
type Server interface {
Resident(ctx context.Context) ([]Resident, error)
Launch(ctx context.Context, modelID string, level LaunchLevel) error
Evict(ctx context.Context, modelID string) error
}
Server is the orchestrator's view of the running model servers: the resident set it can observe, and the launch and evict it can drive. Launch and evict must be idempotent (launching a running model and evicting an absent one are no-ops), which is what lets a reconcile re-run safely. Launch takes a level: when the recovery policy has decided a model keeps running out of memory or wedging, the controller asks for a smaller footprint, down to a CPU-only run, instead of repeating the launch that failed.
Source Files