router

Documentation

Overview

Package router decides which provider serves a request, in what order, based on each combo's configured strategy — and, once a response comes back, whether it was actually good enough to accept. Three concepts are kept deliberately separate (see DECISIONS.md, "Combos v2"):

1. Routing strategy (this package's Strategy interface): decides WHO to try and in WHAT ORDER — a full ranking, not just a winner.
2. Attempt execution (internal/server/chat.go): actually calls each candidate in ranked order until one is accepted or the chain is exhausted.
3. Response gate (gate.go, stream.go): decides whether a technically- successful response is good enough to end the fallback chain.

Index

• func AffinityKey(req openai.ChatCompletionRequest) string
• func ToRankedProviderInfo(ranked []RankedCandidate) []openai.RankedProviderInfo
• type Candidate
• type ComboInfo
• type GateOutcome
• type RankedCandidate
• type ResponseGate
  • func NewResponseGate(cfg config.ResponseGateConfig) *ResponseGate
  • func (g *ResponseGate) Evaluate(content string, toolCalls []openai.ToolCall, sawTerminal bool) GateOutcome
• type RouteContext
  • func NewRouteContext(comboID string, req openai.ChatCompletionRequest, runID string) RouteContext
• type Router
  • func New(cfg *config.Config, providers map[string]provider.Provider, ...) (*Router, error)
  • func (r *Router) Combos() []ComboInfo
  • func (r *Router) Rank(comboID string, req openai.ChatCompletionRequest, runID string) ([]RankedCandidate, RouteContext, error)
  • func (r *Router) RecordOutcome(comboID string, ctx RouteContext, winner string, ok bool)
  • func (r *Router) Resolve(model string) (string, error)
  • func (r *Router) ResponseGateFor(comboID string) *ResponseGate
  • func (r *Router) StrategyName(comboID string) string
• type ScoreFactor
• type Strategy
• type StreamPeekResult
  • func BoundedPeek(ctx context.Context, src <-chan provider.StreamEvent) StreamPeekResult

Functions

func AffinityKey

func AffinityKey(req openai.ChatCompletionRequest) string

AffinityKey identifies a request's stable prompt prefix: the leading system messages plus the first user message, which is precisely the part that does not change across an agent turn's tool round-trips — the growing tail of tool calls and results is deliberately excluded, since including it would produce a different key on every round-trip and defeat the purpose both prefix-affinity routing and sticky routing use this for (see DECISIONS.md). Moved here from internal/server/chat.go — it's a routing concept, not an HTTP-handler concern, and sticky routing needs the same key.

func ToRankedProviderInfo

func ToRankedProviderInfo(ranked []RankedCandidate) []openai.RankedProviderInfo

ToRankedProviderInfo converts a full ranking to the wire shape sent to clients — a pure, lossless projection (provider ID, score, factors, reasons), never a place where a score gets invented or adjusted.

Types

type Candidate

type Candidate struct {
	Provider provider.Provider
	Stats    telemetry.ProviderStats
	// HalfOpen is true if this candidate's breaker is in its half-open
	// trial state — still eligible (Allow() said yes), but strategies may
	// want to treat it more cautiously than a fully closed breaker.
	HalfOpen bool
	// Priority is this candidate's 1-indexed declared position in the
	// combo's provider list (1 = declared first) — the input to the
	// "priority" scoring factor and to the priority strategy's ordering.
	Priority int
	// QualityHint is the operator-configured 0..1 signal from
	// config.ProviderConfig.QualityHint (0 if never set) — see
	// DECISIONS.md for why this is never inferred.
	QualityHint float64
}

Candidate is one provider eligible to serve a request, decorated with everything a Strategy needs to rank it. By the time a Strategy ever sees a Candidate, the hard constraints — circuit breaker state, capability requirements — have already been applied (see eligibleCandidates); a Strategy only ever ranks providers that are actually usable, never scores its way around a real constraint.

type ComboInfo

type ComboInfo struct {
	ID        string
	Strategy  string
	Providers []string
}

ComboInfo is a read-only summary of a combo, for status/diagnostics surfaces that shouldn't reach into the router's internal state.

type GateOutcome

type GateOutcome struct {
	Accepted bool
	Reason   string // set only when !Accepted
}

GateOutcome is a ResponseGate's verdict on a fully-received response.

type RankedCandidate

type RankedCandidate struct {
	Provider Candidate
	// Score is 0..1 for a scoring strategy; strategies that only order
	// (priority, round-robin, prefix-affinity) leave this at 0 and Factors
	// nil — a zero score there is not a claim about quality, just "this
	// strategy doesn't score."
	Score   float64
	Factors []ScoreFactor
	// Reasons are short tags explaining a ranking decision beyond the raw
	// score — "sticky", "last-known-good", "cache-affinity", "explore".
	Reasons []string
}

RankedCandidate is one Strategy.Rank() output entry: a candidate plus why it landed where it did. Every eligible candidate appears here, not just the ones an attempt executor actually calls — fallback can stop before reaching a lower-ranked candidate, but the full ranking is still useful for explainability (see DECISIONS.md, "the UI never recomputes a score").

type ResponseGate

type ResponseGate struct {
	// contains filtered or unexported fields
}

ResponseGate decides whether a technically-successful response (no transport/HTTP error) is actually good enough to end the fallback chain — deterministic, technical checks only. This is never a mechanism for finding a model willing to ignore another's legitimate refusal: a safety or policy refusal is a valid response and is never rejected by anything here. The gate exists for empty/truncated/masked- error responses, not to second-guess what a model chose to say (see DECISIONS.md, "technical error vs quality rejection").

func NewResponseGate

func NewResponseGate(cfg config.ResponseGateConfig) *ResponseGate

NewResponseGate builds a gate from a combo's response config. A zero value config.ResponseGateConfig accepts everything — gating is opt-in, matching v0's behavior of accepting any technically-successful response.

func (*ResponseGate) Evaluate

func (g *ResponseGate) Evaluate(content string, toolCalls []openai.ToolCall, sawTerminal bool) GateOutcome

Evaluate checks a fully-buffered (or fully peeked-and-replayed, for streaming — see stream.go) response. sawTerminal reports whether the provider ever produced a proper finish signal (StreamEvent.Done) before its channel closed.

type RouteContext

type RouteContext struct {
	// ComboID is the combo being routed — useful for strategies that keep
	// per-combo state (sticky pins, LKGP).
	ComboID string
	// AffinityKey is a stable hash of the request's non-growing prefix
	// (system messages + first user message — see AffinityKey), used by
	// the prefix-affinity strategy and the cache_affinity scoring factor.
	// It is deliberately NOT used for Sticky — see RunKey — because it
	// stays identical across every later user turn in the same
	// conversation, not just within one run.
	AffinityKey string
	// RunKey identifies one agent run: one user turn plus every tool
	// round-trip it causes. This is what Sticky actually pins to. It
	// comes from the caller-supplied openai.RunIDHeader when present
	// (Kram's own daemon always sends one — see gatewayclient.WithRunID);
	// a generic OpenAI-compatible caller that never sends the header gets
	// a best-effort fallback to AffinityKey instead of losing Sticky
	// altogether, at the cost of the same session-wide leak this field
	// exists to fix for Kram's own traffic (see DECISIONS.md, "Sticky is
	// run-scoped, not session-prefix-scoped").
	RunKey string
	// NeedsTools and NeedsImages are hard capability constraints: a
	// candidate lacking one is excluded before scoring ever runs (see
	// candidate.go's eligibleCandidates).
	NeedsTools  bool
	NeedsImages bool
}

RouteContext carries per-request information a Strategy needs beyond the candidate list itself — everything here is either directly read from the incoming request or derived deterministically from it, never randomly generated or fabricated.

func NewRouteContext

func NewRouteContext(comboID string, req openai.ChatCompletionRequest, runID string) RouteContext

NewRouteContext derives a RouteContext from an actual request — the only place capability requirements are decided, so eligibleCandidates' hard filtering and every Strategy always see the same, real picture of what this request needs. runID is whatever the caller sent via openai.RunIDHeader ("" if it sent nothing), already extracted by the HTTP handler — see RouteContext.RunKey for what happens when it's empty.

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router resolves a combo ID to a ranked candidate list for one request.

func New

func New(cfg *config.Config, providers map[string]provider.Provider, breakers *breaker.Registry, tel *telemetry.Registry) (*Router, error)

New builds a Router from config, wiring each combo to already-built provider adapters keyed by ID and constructing its configured Strategy.

func (*Router) Combos

func (r *Router) Combos() []ComboInfo

Combos returns a summary of every configured combo, in the order the providers were declared — the same order a non-scoring strategy's fallback follows.

func (*Router) Rank

func (r *Router) Rank(comboID string, req openai.ChatCompletionRequest, runID string) ([]RankedCandidate, RouteContext, error)

Rank returns the full candidate ranking for one request against combo: hard constraints (circuit breaker, capability) are applied first (see eligibleCandidates), then the combo's configured Strategy ranks whatever's left. The returned RouteContext should be passed to RecordOutcome once the request finishes. runID is the caller's openai.RunIDHeader value ("" if absent) — see NewRouteContext.

func (*Router) RecordOutcome

func (r *Router) RecordOutcome(comboID string, ctx RouteContext, winner string, ok bool)

RecordOutcome tells combo's strategy who actually won a request — only strategies that keep state between requests (the weighted family's sticky/LKGP, the standalone lkgp strategy) act on this; everything else is a no-op. Called by the attempt executor once a request finishes, never predicted ahead of time.

func (*Router) Resolve

func (r *Router) Resolve(model string) (string, error)

Resolve maps a client-supplied "model" value to a combo ID: an exact combo ID match wins, otherwise the configured default combo is used.

func (*Router) ResponseGateFor

func (r *Router) ResponseGateFor(comboID string) *ResponseGate

ResponseGateFor returns combo's configured ResponseGate. Unknown combos get a permissive (accept-everything) gate rather than an error, since callers already validate the combo ID via Resolve/Rank first.

func (*Router) StrategyName

func (r *Router) StrategyName(comboID string) string

StrategyName returns combo's configured strategy name (as written in config — "" and "priority" both mean declared order), for the wire response's Strategy field.

type ScoreFactor

type ScoreFactor = openai.ScoreFactor

ScoreFactor and RankedProviderInfo are the router's own names for the wire types in internal/openai (AttemptInfo travels over the gateway's HTTP boundary, so its shape lives there) — aliased here so router code reads naturally without every file needing to know the wire package.

type Strategy

type Strategy interface {
	Name() string
	Rank(ctx RouteContext, candidates []Candidate) []RankedCandidate
}

Strategy decides which candidates to try and in what order. It always returns a full ranking, never just a winner — the existing fallback behavior depends on having an ordered list to fall through, and an explainability UI depends on seeing where every eligible candidate landed, not just the top pick (see DECISIONS.md, "Combos v2").

Rank is called with an already-filtered candidate list: circuit-open and capability-incompatible providers are never passed in (see candidate.go's eligibleCandidates) — a Strategy implementation never needs to re-check those hard constraints itself.

type StreamPeekResult

type StreamPeekResult struct {
	// Committed is true once a meaningful signal was seen — safe to relay
	// to the client from here on.
	Committed bool
	// Reason is set only when !Committed.
	Reason string
	// Buffered holds every event BoundedPeek already consumed from src —
	// the caller must replay these (in order) to its own client before
	// continuing to read further events from src, whether or not it
	// committed (a rejected attempt's buffered events are simply
	// discarded, never replayed).
	Buffered []provider.StreamEvent
}

StreamPeekResult is what BoundedPeek decided after buffering a small prefix of a provider's stream.

func BoundedPeek

func BoundedPeek(ctx context.Context, src <-chan provider.StreamEvent) StreamPeekResult

BoundedPeek consumes events from src until it sees a meaningful signal (a non-empty text delta, or a terminal event carrying tool calls) or gives up — an error, a stream that closes with no meaningful content, or the idle budget below running out. "Received some bytes" is deliberately not sufficient signal on its own: an empty delta, a role-only opening chunk, or a provider that sends periodic keepalive pings would otherwise look like progress — reasoning fragments are the one exception, treated as real (if not yet committable) progress; see streamPeekIdleTimeout.

This runs *before* a caller commits to streaming a response onward to its own client — once that commitment is made (headers sent), no further fallback is possible, which is exactly the problem this exists to solve (see DECISIONS.md, "Bounded streaming peek").