budget

Documentation

Overview

Package budget holds each gateway key to its own limit (#585): what a key has used in its window is summed from usage.jsonl, the records the Usage page shows, so it survives a restart; a request is let in only while the key has something left after what its requests in flight are expected to use, and those are settled with what the call recorded.

The rules:

• A call counts in the window it started in (its record's time). Windows are calendar ones in local time, or cycles of N days from when the limit was set (access.Limit.Window). A reset (#1509) starts the window again from then: what came before it doesn't count, tokens and cost alike.
• Tokens are a call's uncached input, output and cache writes, and its cache reads when the limit says so; cost is an estimate at the Usage page's prices, and a call with no known price adds none.
• A request refused before any upstream (no model, a refusal like this one) uses nothing; a failed call counts the tokens it recorded.
• A streamed reply is settled when it ends, with the usage its vendor reported; one cut off before that counts what was recorded.
• While a request is in flight it holds a reservation: its body's size in tokens (4 bytes a token) plus the key's mean output per call this window. A request is let in while used + reserved is under the cap, so parallel requests overshoot by about one call.
• A key can have several limits, each in its own window (#2227: 5 hours and a week, as a subscription's): a request is let in only while every one has something left, and its reservation is held in each window.
• Only requests that came with a gateway key are limited: from another computer one is required; on this one a request without one (or with a token that is no key) isn't limited.

Index

• func Append(rec usage.Record)
• func Forget()
• func ModelOf(body []byte) string
• type Limiter
  • func (Limiter) Reserve(who access.Identity, body int64, model string, now time.Time) (func(), gwbudget.Refusal)
• type Refusal
  • func Reserve(who access.Identity, body int64, model string, now time.Time) (release func(), refused *Refusal)
  • func (r *Refusal) Error() string
  • func (r *Refusal) ResetAt() time.Time
  • func (r *Refusal) RetryAfter(now time.Time) int
• type Status
  • func All(k access.Key, now time.Time) []Status
  • func Binding(all []Status) *Status
  • func Of(k access.Key, now time.Time) *Status

Functions

func Append

func Append(rec usage.Record)

Append writes rec to usage.jsonl and, for a key's call, counts it against the key's window. The two happen together so a window read from the file meanwhile counts the call once.

func Forget

func Forget()

Forget drops what is kept in memory, as a restart would.

func ModelOf

func ModelOf(body []byte) string

ModelOf is the model a JSON request body names, "" for none.

Types

type Limiter

type Limiter struct{}

Limiter is Reserve as the gateway's budget point asks it (gwbudget.Limiter); plugins/budget serves it.

func (Limiter) Reserve

func (Limiter) Reserve(who access.Identity, body int64, model string, now time.Time) (func(), gwbudget.Refusal)

type Refusal

type Refusal struct {
	Status
	Key string
}

Refusal is why a key's request was turned away.

func Reserve

func Reserve(who access.Identity, body int64, model string, now time.Time) (release func(), refused *Refusal)

Reserve lets a request of who's in, holding a reservation for it until the release it returns is called, or refuses it. body is the request's size in bytes (-1 unknown) and model the model it named, for a cost cap before the key has a price of its own this window.

func (*Refusal) Error

func (r *Refusal) Error() string

Error says it as a client is told: the key, what it used of which cap, and when it resets.

func (*Refusal) ResetAt

func (r *Refusal) ResetAt() time.Time

ResetAt is when the window resets.

func (*Refusal) RetryAfter

func (r *Refusal) RetryAfter(now time.Time) int

RetryAfter is how long until the window resets, in whole seconds.

type Status

type Status struct {
	Period string `json:"period"`
	// Days is a "days" period's cycle, Hours an "hours" one's
	Days     int       `json:"days,omitempty"`
	Hours    int       `json:"hours,omitempty"`
	Start    time.Time `json:"start"`
	Reset    time.Time `json:"reset"`
	Calls    int       `json:"calls"`
	Tokens   int64     `json:"tokens"`
	Cost     float64   `json:"cost"`
	Unpriced int       `json:"unpriced,omitempty"`
	// InFlight requests, and the tokens and cost they are reserved at
	InFlight     int     `json:"inFlight,omitempty"`
	Reserved     int64   `json:"reserved,omitempty"`
	ReservedCost float64 `json:"reservedCost,omitempty"`
	TokenLimit   int64   `json:"tokenLimit,omitempty"`
	CostLimit    float64 `json:"costLimit,omitempty"`
	CacheReads   bool    `json:"cacheReads,omitempty"`
	// TokensLeft and CostLeft are what is left of each cap, 0 at most
	TokensLeft int64   `json:"tokensLeft"`
	CostLeft   float64 `json:"costLeft"`
	// Spent: the key is refused until Reset
	Spent bool `json:"spent"`
}

Status is a key's limit and what it has used of it.

func All

func All(k access.Key, now time.Time) []Status

All is key k's status now of each of its limits (#2227), in the order k keeps them; none for a key with no limit. A window two limits share is read once.

func Binding

func Binding(all []Status) *Status

Binding is the one of a key's statuses that holds it most: of those spent, the one that resets last, as the key is refused until then; else the one with the most of a cap used. Nil for none.

func Of

func Of(k access.Key, now time.Time) *Status

Of is key k's status now of its first limit, nil when it has no limit. It reads the file afresh rather than the gateway's sums, so it is right for a gateway served by another process too.