Documentation
¶
Overview ¶
Package routes is the gateway's router strategy point (P3 of docs/subsystems/plugin-runtime.md): how one pool of routes — a provider's accounts or keys, or the heads of a group's members — is ordered for a request. The gateway lists the pool, rests, affinity and failover itself; a Strategy only says the order.
Smart, the default, is the core's. The others a provider or group can be routed by (order, rotate, usage, pace, weight) are here for a plugin to register: plugins/rotation does, and off, a pool routed by one of them is tried in its listed order, as an unknown routing always was.
Index ¶
- Constants
- Variables
- func Apply(s Strategy, pool string, seats []Seat) (order []int)
- func First(best, n int) []int
- func Forget(pools ...string)
- func MoveRound(pool, picked, kept string)
- func Register(routing string, s Strategy) (remove func())
- func Round(pool string, keys []string, weights []int, rests []bool) int
- func Served(routing string) bool
- type Seat
- type Strategy
Constants ¶
const ( )
Shares of an allowance past which an account is kept for when the others can't take a request: low, and all but used up — where the agent signed in to it is signed in to another, too (provider.SpentShare).
Variables ¶
var Builtin = map[string]Strategy{ provider.Ordered: Ordered, provider.Rotate: Rotate, provider.LeastUsed: LeastUsed, provider.Pace: Pace, provider.Weighted: Weighted, provider.FreeFirst: FreeFirst, }
Builtin are the routings besides Smart and their strategies, as plugins/rotation registers them.
var FreeFirst = Strategy{ByFree: true, Order: func(_ string, seats []Seat) []int { free, paid := make([]int, 0, len(seats)), make([]int, 0, len(seats)) for i, s := range seats { if s.Free { free = append(free, i) } else { paid = append(paid, i) } } return append(free, paid...) }}
FreeFirst spends what costs nothing before what is paid for: the seats whose model its vendor serves at no cost to the subscription's allowance or credits go first, in their listed order, and the rest after them, also in their listed order. So a request whose model is free somewhere on the pool uses that first, and only when those can't answer does it go to one that spends credits or quota — nothing is left unused that cost nothing.
It orders one tier only, unlike Smart: a free seat is not pushed behind a paid one for running low, since the free one spends nothing by running out of. Among the paid, the listed order stands rather than Smart's renewal-weighted one, because what this routing is for is the free tier; set a provider's own routing (Smart, in order, weekly pace) if the paid seats should be spent smartly among themselves — a provider or group in this group keeps its own routing either way.
A seat whose model costs nothing nowhere is not free: it goes with the paid ones, so a routing set on a pool without free models behaves as In order.
var LeastUsed = Strategy{ByUse: true, Order: func(_ string, seats []Seat) []int { idx := identity(len(seats)) sort.SliceStable(idx, func(a, b int) bool { sa, sb := seats[idx[a]], seats[idx[b]] if ta, tb := tier(sa.Used), tier(sb.Used); ta == 2 && tb == 2 && sa.Credits != sb.Credits { return sa.Credits } if sa.Used != sb.Used { return sa.Used < sb.Used } return sa.Tokens < sb.Tokens }) return idx }}
LeastUsed puts a subscription, or a key that reads its own windows, by the share of its allowance used, as the vendor says; then, and for other keys, by what magpie sent it lately. Of those all but used up, one that goes on on its credits goes first (spentBefore).
var Ordered = Strategy{Exact: true, Order: func(_ string, seats []Seat) []int { return identity(len(seats)) }}
Ordered tries a pool in its order: the first until it can't answer.
var Pace = Strategy{ByUse: true, Order: func(_ string, seats []Seat) []int { // the bands of pace alike within a tenth, the highest first: drawn // before sorting, from each band's highest down, so that sorting by // them is consistent — "within a tenth of each other" alone is not, // three paces a twelfth apart each going round in a circle. Drawn among // those the pace orders alone — one running low would otherwise set a // band's top and part two alike behind it var byPace []int for i, s := range seats { if tier(s.Used) == 0 && !s.Learns { byPace = append(byPace, i) } } sort.SliceStable(byPace, func(a, b int) bool { return seats[byPace[a]].Pace > seats[byPace[b]].Pace }) band, top := make([]int, len(seats)), 0.0 for k, i := range byPace { if p := seats[i].Pace; k == 0 || p < 0.9*top { top = p if k > 0 { band[i] = band[byPace[k-1]] + 1 continue } } else { band[i] = band[byPace[k-1]] } } idx := identity(len(seats)) sort.SliceStable(idx, func(a, b int) bool { sa, sb := seats[idx[a]], seats[idx[b]] if ta, tb := tier(sa.Used), tier(sb.Used); ta != tb { return ta < tb } else if ta == 2 && sa.Credits != sb.Credits { return sa.Credits } else if ta > 0 { if sa.Used != sb.Used { return sa.Used < sb.Used } } else if sa.Learns != sb.Learns { return sa.Learns } else if band[idx[a]] != band[idx[b]] { return band[idx[a]] < band[idx[b]] } return sa.Tokens < sb.Tokens }) return idx }}
Pace puts a subscription by what it has left of its week per hour until that renews, the most first: it has the most to lose at the reset, where the most left alone (least used) sends a fresh account ahead of one with 80% left and an hour to go, and the soonest reset alone (smart) sends one with 3% left and an hour to go ahead of one with 90% and two. The tiers are Smart's: one at 90% or more of any window that counts the model — the five hours too, a rate cap it would be sent into before its allowance was read again — waits until the others can't answer, one all but used up until nothing else can; each tier by the share used, one all but used up that goes on on its credits first (spentBefore). Among the rest, one not known that tells what it has left as it answers goes first while it isn't, as in Smart, else it would never be known; then the pace, those alike within a tenth by what magpie sent them lately, then in their order, keeping the vendor's prompt cache warm. An account not known counts as a whole week ahead of it; a key has no week, unless it reads its own windows (a sub2api key's limits). A Codex account that spends a reset about to run out by itself goes by the hours until then when that is sooner (Allowance.Pace, #717).
var Rotate = Strategy{Order: func(pool string, seats []Seat) []int { turns.Lock() n := turns.m[pool] % len(seats) turns.m[pool]++ turns.Unlock() return append(identity(len(seats))[n:], identity(n)...) }}
Rotate starts each request one further along the pool.
var Smart = Strategy{Order: func(_ string, seats []Seat) []int { var fine, low, spent []int for i, s := range seats { switch tier(s.Used) { case 2: spent = append(spent, i) case 1: low = append(low, i) default: fine = append(fine, i) } } sort.SliceStable(fine, func(i, j int) bool { a, b := seats[fine[i]], seats[fine[j]] if a.Learns != b.Learns { return a.Learns } ri, rj := a.Soon, b.Soon for k := 0; k < len(ri) || k < len(rj); k++ { var x, y time.Time // to the hour, so a few minutes don't reorder if k < len(ri) { x = ri[k].Truncate(time.Hour) } if k < len(rj) { y = rj[k].Truncate(time.Hour) } switch { case x.Equal(y): continue case x.IsZero() || y.IsZero(): return y.IsZero() } return x.Before(y) } return false }) sort.SliceStable(low, func(i, j int) bool { return seats[low[i]].Used < seats[low[j]].Used }) sort.SliceStable(spent, func(i, j int) bool { return spentBefore(seats[spent[i]], seats[spent[j]]) }) return append(append(fine, low...), spent...) }}
Smart is the default: of those with quota to spare, the one whose allowance renews soonest, since what it has left is lost then, while one renewing later keeps: the biggest window decides — the week, not the five hours in it — and the next one only when that renews in the same hour. An account with no week (Claude Enterprise's five hours alone) goes by its five hours, so ahead of every week but one renewing sooner; a window not started renews its whole span from now (#576). A Codex account that spends a reset about to run out by itself (auto-use on, its windows used) renews when that is spent, if sooner: what it has left is lost there just the same (#718). Only holding one, or one that runs out after the week renews, changes nothing. Those alike stay in their order, keeping the vendor's prompt cache warm. Past that, whichever has the most left, and one all but used up only when nothing else can take it, one that goes on on its credits first of those (spentBefore). Only the windows that count the model do: Opus's own weekly allowance being used up leaves Sonnet alone.
One not known that tells what it has left as it answers goes first, once: else, kept after those known, it would never answer and never be known (Anthropic's usage endpoint can turn magpie away for hours).
var Weighted = Strategy{Order: func(pool string, seats []Seat) []int { keys, weights, rests := make([]string, len(seats)), make([]int, len(seats)), make([]bool, len(seats)) for i, s := range seats { keys[i], weights[i], rests[i] = s.Key, max(s.Weight, 1), s.Resting } return First(Round(pool, keys, weights, rests), len(seats)) }}
Weighted spreads a pool's requests over its seats by each one's weight, smoothly (Round): the one the pool's round picks now first, the others after it in their order, for failover.
Functions ¶
func Apply ¶
Apply is the order s gives seats, checked: an order that isn't a permutation of them, or a strategy that panics, leaves them in their listed order, logged. A strategy only orders, so failing it fails open: every route is still tried.
func First ¶
First is the order of n with best first and the others after it in their order; best below 1 leaves it as it is.
func Forget ¶
func Forget(pools ...string)
Forget drops what pool's round and turn hold, every pool's with none named: a test starts each anew.
func MoveRound ¶
func MoveRound(pool, picked, kept string)
MoveRound charges pool's last pick to kept rather than picked, when affinity kept a conversation with kept: the round counts the requests each really took, so new conversations make up the share.
func Register ¶
Register serves routing by s until remove is called. Registering one already served stacks: the last serves until it is removed.
Types ¶
type Seat ¶
type Seat struct {
// Key is who it is: the account or key's rest key, the same for the
// same route from request to request
Key string
// Account is a subscription's account, not an API key
Account bool
// Known: its allowance is known. Used, Soon and Pace are read from it
Known bool
// Learns: an account not known yet, of an agent that tells what it has
// left as it answers (Claude Code's rate_limit_event)
Learns bool
// Used is the share of its allowance used that counts the model, 0–100
Used float64
// Soon is when its windows renew, the biggest first (Allowance.Renewal)
Soon []time.Time
// Pace is its week left per hour until that renews; an account not
// known has a whole week (provider.FreshPace), a key none
Pace float64
// Weight is its key's weight, at least 1
Weight int
// Resting: it rests after a failure now
Resting bool
// Tokens is what magpie sent it lately, decayed by an hour's half-life
Tokens float64
// Free: the model this seat would be asked for costs nothing on it —
// its vendor serves that model at no cost to the subscription's
// allowance or credits (catalog.Model's Free). FreeFirst goes by it.
Free bool
// Credits: a Codex account that goes on past its allowance on its
// credits, and is let spend them. All but used up, it goes before
// those used up with nothing to go on with.
Credits bool
}
Seat is one route of a pool, as a strategy weighs it.
type Strategy ¶
type Strategy struct {
// Order is the order to try seats in: a permutation of their indexes.
// pool names the pool (a provider's id, or "group/<id>"), for a
// strategy that keeps a round per pool. It is asked with two seats or
// more.
Order func(pool string, seats []Seat) []int
// Exact: its order stands as it says. Otherwise, of a provider's API
// keys, those that fit the request better go first, each kept in its
// order among those alike.
Exact bool
// ByUse: it weighs by Tokens, which the routing trace then shows
ByUse bool
// ByFree: it weighs by Free, so the gateway works out which seat's
// model costs nothing there, and the routing trace then shows it.
// Only when set: working it out reads each provider's model list.
ByFree bool
}
Strategy orders a pool.
Source Files