fairshare

Documentation

Overview

Package fairshare decides the vLLM priority of every request so that a GPU pool shared by several teams behaves like a reservation with borrowing:

• each team owns share × C of the pool's sustainable concurrency C;
• a request that keeps its team under that reservation is sent "within share" (priority 0 for interactive, 1 for batch);
• a request above it "borrows" idle capacity (priority 10 / 11 and more), so it runs when nobody else needs the GPUs, and it is the one vLLM preempts when they do.

The mapping relies on two properties of vLLM's priority scheduler (vllm/v1/core/sched/scheduler.py and request_queue.py, opened 2026-10-23): the waiting queue is a min-heap on (priority, arrival_time), and when a running request runs out of KV blocks vLLM preempts the running request with the largest (priority, arrival_time). A waiting request never preempts a running one.

Among borrowers, the order follows Slurm's Fair Tree level fairshare LF = S / U (https://slurm.schedmd.com/fair_tree.html): S is the team's share normalised over the teams of the pool, U its decayed usage normalised the same way. Usage is the consumption (cost units) of the team's finished requests, decayed with a half-life as Slurm's PriorityDecayHalfLife does (priority_multifactor.c applies usage_raw *= decay_factor^Δt with decay_factor = 1 - 0.693/half_life; this package uses the exact exponential exp(-ln2·Δt/half_life), of which Slurm's factor is the first-order expansion). The most over-served borrower gets the largest priority number, so it is admitted last and preempted first.

Index

• Constants
• func Apportion(capacity int, names []string, shares map[string]float64) map[string]int
• type Class
• type Grant
  • func NewRemoteGrant(teamName string, priority int, borrowing, saturated bool, ...) *Grant
  • func (g *Grant) Release(consumed float64)
  • func (g *Grant) Team() string
• type Pool
  • func NewPool(name string, capacity int, shares map[string]float64, halfLife time.Duration, ...) *Pool
  • func (p *Pool) Acquire(teamName string, class Class, estTokens int64) *Grant
  • func (p *Pool) AddPhantoms(teamName string, n int, until time.Time)
  • func (p *Pool) Adopt(teamName string, priority int, borrowing bool, estTokens int64) *Grant
  • func (p *Pool) Capacity() int
  • func (p *Pool) Charge(teamName string, consumed float64)
  • func (p *Pool) DropPhantom(teamName string) bool
  • func (p *Pool) Lambda() float64
  • func (p *Pool) Name() string
  • func (p *Pool) Reserved() map[string]int
  • func (p *Pool) SetCapacity(capacity int) int
  • func (p *Pool) SetUsage(teamName string, usage float64)
  • func (p *Pool) Snapshot() []TeamStat
  • func (p *Pool) TryAcquire(teamName string, class Class, estTokens int64) (*Grant, bool)
  • func (p *Pool) Usage() (usage map[string]float64, inflight map[string]int)
• type TeamStat

Constants

const (
	// WithinShareBase is the priority of a request inside its team's
	// reservation, plus the class.
	WithinShareBase = 0
	// BorrowBase is the priority of the least over-served borrower, plus
	// the class.
	BorrowBase = 10
	// RankStep separates borrowers of successive fairshare ranks.
	RankStep = 10
	// MaxRank bounds the rank so priorities stay small.
	MaxRank = 9
)

Priority layout (lower runs earlier in vLLM).

Functions

func Apportion

func Apportion(capacity int, names []string, shares map[string]float64) map[string]int

Apportion divides the C slots of a pool into integer reservations with the largest-remainder (Hamilton) method: every team first gets floor(share × C), then the slots left up to floor(Σ share × C) go one each to the largest fractional parts (ties broken by name order). The sum never exceeds C, and equals C when the shares sum to 1. Rounding each quota up instead (the first version) could promise more within-share slots than C: with C = 10 and 0.8125 / 0.1875 it gave 9 + 2 = 11, so one of B's guaranteed requests could wait in vLLM's queue at equal priority behind A's ninth.

Types

type Class

type Class int

Class is the service class of a request.

const (
	// Interactive is latency sensitive.
	Interactive Class = 0
	// Batch is throughput oriented and yields to interactive work of the
	// same standing.
	Batch Class = 1
)

type Grant

type Grant struct {
	Priority  int
	Borrowing bool
	// Saturated: when this request arrived, the pool already had C or more
	// requests in flight (the gateway's own view of saturation, #58).
	Saturated bool
	// contains filtered or unexported fields
}

Grant is the decision for one request. Release must be called exactly once when the request has finished (successfully or not).

func NewRemoteGrant

func NewRemoteGrant(teamName string, priority int, borrowing, saturated bool, release func(consumed float64)) *Grant

NewRemoteGrant returns a grant decided elsewhere (by the replica that owns the pool, package ha). Release calls release once, with the consumption; the grant is not counted in any local Pool.

func (*Grant) Release

func (g *Grant) Release(consumed float64)

Release ends the request and charges the resources it actually consumed (cost units, e.g. GPU-hours; negative or NaN values count as 0) to the team's usage. It is idempotent: only the first call counts.

Usage is charged from consumption, never from in-flight time: a request waiting in vLLM's queue is in flight for the gateway but consumes nothing, and charging it made a flooding team look as consumed as the team it was starving (TestSaturationConvergesToShares caught it).

func (*Grant) Team

func (g *Grant) Team() string

Team returns the team name of the grant.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool tracks one capacity domain.

func NewPool

func NewPool(name string, capacity int, shares map[string]float64, halfLife time.Duration, now func() time.Time) *Pool

NewPool returns a pool of capacity C with the given team shares. A halfLife of 0 disables decay (usage accumulates forever). now may be nil for time.Now.

func (*Pool) Acquire

func (p *Pool) Acquire(teamName string, class Class, estTokens int64) *Grant

Acquire records a new in-flight request of a team and returns its priority. A team unknown to the pool has a share of 0: it can only borrow.

func (*Pool) AddPhantoms

func (p *Pool) AddPhantoms(teamName string, n int, until time.Time)

AddPhantoms counts n requests of a team as in flight until the given time, without grants: requests that may still run but that nobody can release (their replica died, or a previous owner's in-flight count).

func (*Pool) Adopt

func (p *Pool) Adopt(teamName string, priority int, borrowing bool, estTokens int64) *Grant

Adopt counts a request decided before (by a previous owner of the pool, or admitted without one) as in flight with the standing it was given, and returns its grant. Nothing is decided: the priority is the one the request already runs with.

func (*Pool) Capacity

func (p *Pool) Capacity() int

Capacity returns C.

func (*Pool) Charge

func (p *Pool) Charge(teamName string, consumed float64)

Charge adds consumption to a team's usage without a grant (the release of a request this pool never counted, package ha).

func (*Pool) DropPhantom

func (p *Pool) DropPhantom(teamName string) bool

DropPhantom removes the team's phantom that would last longest and reports whether there was one: a request it stood for turned out to be a live grant after all.

func (*Pool) Lambda

func (p *Pool) Lambda() float64

Lambda is the decay rate ln2/half-life in 1/s (0: no decay).

func (*Pool) Name

func (p *Pool) Name() string

Name returns the pool name.

func (*Pool) Reserved

func (p *Pool) Reserved() map[string]int

Reserved returns the within-share slots of each organisation.

func (*Pool) SetCapacity

func (p *Pool) SetCapacity(capacity int) int

SetCapacity changes C (an elastic pool whose instances come and go, #75) and recomputes the within-share reservations with Apportion. Requests in flight keep their within-share or borrowing status; new requests are judged against the new reservations (and, for TryAcquire, the new C). It returns the previous C.

func (*Pool) SetUsage

func (p *Pool) SetUsage(teamName string, usage float64)

SetUsage sets a team's decayed usage (restored from a previous owner). Negative, NaN and infinite values count as 0.

func (*Pool) Snapshot

func (p *Pool) Snapshot() []TeamStat

Snapshot returns the current state of every team, in a stable order.

func (*Pool) TryAcquire

func (p *Pool) TryAcquire(teamName string, class Class, estTokens int64) (*Grant, bool)

TryAcquire is Acquire only when the pool has room (fewer than C requests in flight), decided and recorded atomically: a fallback that moves a request here never pushes this pool past C, so two saturated pools cannot hand requests to each other (#58).

func (*Pool) Usage

func (p *Pool) Usage() (usage map[string]float64, inflight map[string]int)

Usage returns every team's decayed usage and in-flight count (phantoms included), decayed to now: what an owner persists for the next one.

type TeamStat

type TeamStat struct {
	Team           string
	Share          float64
	Reserved       int // within-share slots, see Apportion
	InFlight       int
	Borrowing      int
	InFlightTokens int64
	Usage          float64
	LevelFairshare float64
	Rank           int
	// Phantoms are requests counted in flight without a grant (package ha:
	// a failover's safety margin, a silent replica's requests); InFlight
	// does not include them.
	Phantoms int
}

TeamStat is a snapshot of one team in a pool.