agilelimiter

README

agileLimiter

中文文档

Adaptive rate limiting for outbound dependencies, and adaptive concurrency limiting for inbound services.

agileLimiter watches latency and adjusts how much traffic to allow. It is a small, zero-dependency Go module. You give it a ceiling and a safe floor; it tries to use the headroom above the floor without overwhelming the other side.

• Outbound: pace calls to a dependency with Limiter.
• Inbound: cap concurrent work in your own service with ConcurrencyLimiter.
• Safe by default: normal traffic never goes below FloorRate; an outage can fall to the lower ProbeRate.

Install

go get github.com/Yiming1997/agilelimiter

Requires Go 1.24 or newer. No external dependencies.

Outbound: call a dependency

import (
    "time"

    "github.com/Yiming1997/agilelimiter"
)

lim := agilelimiter.New(agilelimiter.Config{
    MaxRate:   500,                     // hard ceiling
    FloorRate: 250,                     // normal floor: a rate you trust
    ProbeRate: 10,                      // outage floor: keep probing, don't hammer
    SLOTarget: 200 * time.Millisecond,  // optional; your latency red line
})
lim.Start()
defer lim.Stop()

// Do takes a slot, times the call, and feeds the result back to the controller.
err := lim.Do(func() error {
    return callDownstream(ctx)
})

MaxRate is the only required field. Start with MaxRate; add FloorRate when you know the static rate you trust, and ProbeRate when you want outages to back off further.

HTTP client

import "github.com/Yiming1997/agilelimiter/httpclient"

client := &http.Client{
    Transport: httpclient.Wrap(http.DefaultTransport, lim),
}

httpclient.Wrap paces each request, measures the round trip, and reports it to the limiter. Use httpclient.WithVerdict when some responses should not count as capacity signals (cache hits, redirects, cheap errors, and so on).

Manual Take / Observe

Use this when the call is not a plain synchronous function:

lim.Take()
start := time.Now()
err := callDownstream(ctx)
lim.ObserveOutcome(time.Since(start), err)

If the call never reached the downstream, return agilelimiter.Unreachable from Config.Classify (or call lim.Unavailable() on the manual path). Do not count a refused connection as a fast success.

Inbound: protect your own service

For inbound traffic, the useful knob is concurrency, not rate: clients decide how fast requests arrive, while you decide how many run at once.

lim := agilelimiter.NewConcurrency(agilelimiter.ConcurrencyConfig{
    MaxLimit:   500, // hard ceiling
    FloorLimit: 100, // normal floor
    ProbeLimit: 10,  // outage floor
    SLOTarget:  200 * time.Millisecond,
})
lim.Start()
defer lim.Stop()

err := lim.Do(func() error {
    return handleRequest(w, r)
})
if errors.Is(err, agilelimiter.ErrOverloaded) {
    http.Error(w, "overloaded", http.StatusTooManyRequests)
}

Admission is non-blocking: at the limit, Do returns ErrOverloaded immediately instead of building a queue. Use TryTake/Release if you need to hold a slot across multiple calls.

HTTP server middleware

import "github.com/Yiming1997/agilelimiter/httpserver"

root := http.NewServeMux()
root.Handle("/debug/limiter", httpserver.StatsHandler(lim))
root.Handle("/", httpserver.Wrap(mux, lim,
    httpserver.WithRetryAfter(time.Second)))

http.ListenAndServe(":8080", root)

httpserver.Wrap sheds with 429 when the limit is reached and records only admitted requests. Keep StatsHandler outside the wrapped subtree so it remains readable while the service is saturated.

Configuration

Most services only need these fields:

Field	Meaning
MaxRate / MaxLimit	Hard ceiling. Required.
FloorRate / FloorLimit	Normal operating floor. Keep it near the static rate you trust.
ProbeRate / ProbeLimit	Floor while the downstream is unreachable. Keep it low.
SLOTarget	Optional latency target. Enables the tail guard and the SLO anchor.
Classify	Which errors mean "never reached the downstream".

MinRate / MinLimit still work as a compatibility fallback: if FloorRate or ProbeRate is zero, it inherits the legacy value.

Presets are optional shortcuts:

Preset: agilelimiter.PresetLatencyFirst

Start with the zero preset and explicit MaxRate/FloorRate; add a preset only when its policy matches what you want.

How it behaves

On every tick, the controller looks at the median and tail latency and adjusts the allowed rate or concurrency:

• latency is healthy: try a little more
• latency is above the target window: back off
• latency is in between: hold steady
• tail latency is above the SLO: back off

It keeps a slow baseline of normal latency, so a temporary spike does not immediately rewrite the target. When the downstream is unreachable, the rate can fall to ProbeRate; a normal response brings it back to the regular floor.

There are no formulas to tune here. If you need the control law and the measured trade-offs, read docs/design.md.

Observability

log.Printf("rate=%.0f stats=%+v", lim.Rate(), lim.Stats())
log.Printf("resolved config=%+v", lim.Config())

Export at least:

• Rate() / Limit()
• Stats().Samples — zero means the controller is blind
• Stats().P50 and Stats().P99
• Config() — the defaults and presets actually in effect

Things to know

• This is not a circuit breaker. Pair it with one if “stop calling entirely” is part of your failure policy.
• It is per process. Multiple instances each learn and limit independently.
• It does not queue. Inbound overload is refused, not buffered.
• It does not distribute quotas across machines.
• FloorRate is a capacity bet. If real capacity drops below it, the floor is no longer safe; use ProbeRate for actual outages and alert on sustained floor operation.
• Measure the downstream round trip, not the time spent waiting inside the limiter. Do does this for you.

Examples and tests

go run ./example                 # outbound
go run ./example/concurrency     # inbound
go run ./example/httpserver      # HTTP middleware

go test ./...
go test -race ./...

More detail:

• docs/design.md — controller design and trade-offs
• docs/integrations.md — HTTP/gRPC integration notes
• httpclient and httpserver — drop-in adapters
• sim/ — virtual-clock simulations and comparisons

License

MIT. See LICENSE.

Documentation

Overview

Package agilelimiter implements an adaptive, latency-driven rate limiter that protects a downstream dependency.

Design in one sentence: a simple leaky bucket acts as the *actuator* (it paces outbound requests at whatever rate it is told), while a feedback controller decides *what that rate should be*, based on the downstream's observed latency. A static ceiling is enforced in two independent places, so the controller can never exceed the value you are willing to promise.

Control loop (per tick):

p50 (per-tick histogram)  ─┐
slow baseline (one EWMA)  ─┼─> target window ─> AIMD / hold ─> clamp ─> bucket.SetRate
p99 (per-tick histogram)  ─┘   (baseline)       + p99 SLO guard

The tick's median is compared against a window: with an SLOTarget, by default, 0.8 x SLOTarget; without one, (1+TargetInflation) x the slow EWMA baseline. While the median stays under it the controller probes the rate up; once it rises past it the controller backs off by a fixed step. Operating with some queuing — rather than backing off at the first sign of it — is what keeps utilization reasonable. The tail guard, by default, backs off when more than 5% of a tick is above the SLO by more than its statistical margin.

Measurement discipline (important): feed Observe the *downstream* round-trip time — measured from the moment the request is actually sent, NOT from the moment it was planned. Feeding end-to-end latency that includes the limiter's own throttle wait creates a self-reinforcing feedback loop (backing off raises the measured latency, which causes more backing off) and collapses the rate.

The planned-send timestamp returned by Take() is still useful — use it to build a coordinated-omission-safe end-to-end metric for SLO reporting:

t0 := lim.Take()                 // planned send time (blocks until granted)
start := time.Now()              // actually sent
resp := doRequest(ctx)
lim.Observe(time.Since(start))   // downstream RTT -> drives the controller
_ = time.Since(t0)               // CO-safe latency -> reporting only

Index

• Variables
• func Do[T any](l *Limiter, fn func() (T, error)) (T, error)
• type Anchor
• type Clock
• type ConcurrencyConfig
  • func (cc ConcurrencyConfig) Validate() error
• type ConcurrencyLimiter
  • func NewConcurrency(cc ConcurrencyConfig) *ConcurrencyLimiter
  • func (l *ConcurrencyLimiter) Config() Resolved
  • func (l *ConcurrencyLimiter) Do(fn func() error) error
  • func (l *ConcurrencyLimiter) InFlight() int
  • func (l *ConcurrencyLimiter) Limit() int
  • func (l *ConcurrencyLimiter) Observe(d time.Duration)
  • func (l *ConcurrencyLimiter) ObserveOutcome(d time.Duration, err error)
  • func (l *ConcurrencyLimiter) ObserveTruncated(d time.Duration)
  • func (l *ConcurrencyLimiter) Release()
  • func (l *ConcurrencyLimiter) Start()
  • func (l *ConcurrencyLimiter) Stats() Stats
  • func (l *ConcurrencyLimiter) Stop()
  • func (l *ConcurrencyLimiter) TryTake() bool
  • func (l *ConcurrencyLimiter) Unavailable()
• type Config
  • func (c Config) Validate() error
• type Limiter
  • func New(cfg Config) *Limiter
  • func (l *Limiter) Config() Resolved
  • func (l *Limiter) Do(fn func() error) error
  • func (l *Limiter) Observe(rtt time.Duration)
  • func (l *Limiter) ObserveOutcome(d time.Duration, err error)
  • func (l *Limiter) ObserveTruncated(d time.Duration)
  • func (l *Limiter) Rate() float64
  • func (l *Limiter) Start()
  • func (l *Limiter) Stats() Stats
  • func (l *Limiter) Stop()
  • func (l *Limiter) Take() time.Time
  • func (l *Limiter) Unavailable()
• type Outcome
• type Preset
  • func (p Preset) String() string
• type Resolved
• type Stats

Variables

var ErrOverloaded = errors.New("agilelimiter: concurrency limit reached")

ErrOverloaded is returned by ConcurrencyLimiter.Do when the concurrency limit is already reached. Map it to HTTP 429 (StatusTooManyRequests) or gRPC codes.ResourceExhausted.

Functions

func Do

func Do[T any](l *Limiter, fn func() (T, error)) (T, error)

Do is the generic form of Limiter.Do, for calls that also return a value:

resp, err := agilelimiter.Do(lim, func() (*http.Response, error) {
	return client.Do(req)
})

Types

type Anchor

type Anchor int

Anchor selects what the congestion threshold is measured against.

const (
	// AnchorUnset is the zero value: let Preset decide. PresetNone picks
	// AnchorSLO when SLOTarget is set and AnchorBaseline when it is not.
	//
	// It exists because AnchorBaseline has to be a real, choosable value — so the
	// zero value cannot also mean "baseline" without making "the caller did not
	// say" indistinguishable from "the caller asked for the baseline", and then
	// a preset could never switch an unsuspecting config to the SLO anchor.
	AnchorUnset Anchor = iota

	// AnchorBaseline is the relative controller: back off when recent latency
	// rises above (1+TargetInflation) x the *fastest latency this downstream has
	// been seen to produce*. It protects the knee of the latency curve and needs
	// no SLO, but it has no notion of how much SLO headroom is left: a downstream
	// answering in 20ms that degrades to 50ms is 2.5x its own baseline, so the
	// relative law backs off to MinRate even though 50ms may be a quarter of the
	// SLO, and it then stays there for minutes (MinRate -> MaxRate at UpStep per
	// tick). It is the law without an SLOTarget, or with PresetBalanced. See
	// docs/design.md §9.
	AnchorBaseline

	// AnchorSLO is the absolute controller: the backoff threshold is
	// sloWindowFraction x SLOTarget and nothing else, so a latency rise that
	// stays well inside the SLO costs nothing. The healthy baseline is still
	// tracked (Stats, outage anchoring) but it never decides a backoff.
	//
	// The trade-off is the mirror image: it keeps probing up while the SLO has
	// headroom, so it cannot find a knee that sits below the SLO. Use it when the
	// SLO is the contract and the downstream degrades gracefully until it, which
	// is the usual case for a dependency with a published latency target.
	//
	// Requires SLOTarget > 0; with no SLO configured it degrades to
	// AnchorBaseline.
	AnchorSLO
)

type Clock

type Clock interface {
	Now() time.Time
	Sleep(time.Duration)
}

Clock is the minimal time source the limiter needs. Injecting it keeps the package testable with a virtual (fake) clock.

type ConcurrencyConfig

type ConcurrencyConfig struct {
	// Preset fills the policy fields you leave at zero (see Config.Preset).
	Preset Preset

	// MinLimit is the legacy fallback for FloorLimit and ProbeLimit.
	// FloorLimit is the normal operating floor; ProbeLimit is the floor while
	// the service is unreachable. MaxLimit is the hard ceiling.
	MinLimit   int
	FloorLimit int
	ProbeLimit int
	MaxLimit   int
	// InitialLimit is the starting concurrency. Cold-start low and let the
	// controller ramp up — starting at MaxLimit risks flooding yourself before
	// the first feedback arrives.
	InitialLimit int

	// SLOTarget enables the tail-latency guard: when the per-tick p99 exceeds
	// it, the controller forces a full backoff step. Zero disables the guard.
	SLOTarget time.Duration

	// Anchor chooses what the backoff threshold is measured against. See
	// Config.Anchor: AnchorSLO measures against the SLO itself, so a benign rise
	// in handling time that stays inside it costs no concurrency.
	Anchor Anchor

	// TailFactor scales SLOTarget for the p99 guard only (see Config.TailFactor).
	TailFactor float64
	// TailTolerance switches the guard to a ratio test (see Config.TailTolerance).
	TailTolerance float64

	// BackoffTicks and RecoverTicks are the time hysteresis (see Config).
	BackoffTicks int
	RecoverTicks int

	// SlowHalfLife is the healthy-baseline EWMA half-life. SlowDownHalfLife
	// controls how quickly it follows improvements down.
	SlowHalfLife     time.Duration // healthy baseline (creeps up slowly)
	SlowDownHalfLife time.Duration // baseline follows improvements down

	// MaxBaselineRise caps how fast the healthy baseline may creep upward, as a
	// fraction per tick. Zero takes the default. See Config.MaxBaselineRise.
	MaxBaselineRise float64

	// Tick is the control-loop period.
	Tick time.Duration

	// TargetInflation is how far above the healthy baseline latency the fast
	// signal may rise before the controller backs off. 1.0 means "accept up to
	// 2x the baseline".
	TargetInflation float64
	// UpStep / DownStep cap the fractional change per tick (down faster than up).
	UpStep   float64
	DownStep float64
	// SlowStartGain is the multiplicative increase used while the controller has
	// never had to back off, i.e. while it does not know where the knee is.
	// Default 2.0: reaching MaxLimit costs log2 of the ratio in ticks instead of
	// ln(ratio)/ln(1+UpStep). Set 1 for pure AIMD. See Config.SlowStartGain.
	SlowStartGain float64
	// Deadband is the hold band around the target latency, as a fraction of it.
	Deadband float64

	// Classify maps a failed request to what the controller is told about it. It
	// is only consulted for non-nil errors; a nil error always means Reached.
	//
	// The default (nil) treats every error as Reached. That is wrong for a
	// handler that fails without doing any work — a cached 503 while a dependency
	// is down, an auth rejection, a circuit breaker returning early. Its handling
	// time is ~0, the controller reads it as "this service is fast", and it raises
	// the limit while the service is actually degrading. Return Unreachable for
	// those; return Aborted for client-side cancellations.
	Classify func(err error) Outcome

	// Clock defaults to the real clock. Inject a fake one in tests.
	Clock Clock
}

ConcurrencyConfig configures a ConcurrencyLimiter. It mirrors Config, but the bounds are a *concurrency limit* (how many requests may be in flight at once) rather than a rate. The control-law knobs are identical and share the same defaults.

func (ConcurrencyConfig) Validate

func (cc ConcurrencyConfig) Validate() error

Validate reports what NewConcurrency would otherwise repair silently. It is Config.Validate applied to the equivalent rate config, so the rules — and the wording — are identical; read MaxLimit as MaxRate.

type ConcurrencyLimiter

type ConcurrencyLimiter struct {
	// contains filtered or unexported fields
}

ConcurrencyLimiter is an adaptive, self-protecting concurrency limiter for the *inbound* side of a service: it caps how many requests are handled at once and admits or refuses each one non-blockingly.

It shares its brain with Limiter — the same p50 signal, slow baseline, target window and p99 guard — and swaps only the actuator, from a leaky bucket to a semaphore. The signal is the handler's own handling time; the target is a concurrency count instead of a rate.

limit climbs   ->   latency rises past the window   ->   limit backs off

A request that cannot get a slot is refused immediately (ErrOverloaded) rather than queued, which is what keeps the limiter from amplifying an overload.

func NewConcurrency

func NewConcurrency(cc ConcurrencyConfig) *ConcurrencyLimiter

NewConcurrency builds a ConcurrencyLimiter. It does not start the control loop; call Start (or drive ctrl.step yourself, as tests do).

Internally it reuses the outbound controller verbatim: that controller only multiplies a scalar by a factor and clamps it, so it is agnostic to whether the scalar is a rate or a concurrency limit. We map the limit bounds onto its bound fields and point its setter at the semaphore.

func (*ConcurrencyLimiter) Config

func (l *ConcurrencyLimiter) Config() Resolved

Config returns the configuration this limiter is actually running with. The MaxRate/MinRate/InitialRate fields hold the concurrency bounds; Kind says so.

func (*ConcurrencyLimiter) Do

func (l *ConcurrencyLimiter) Do(fn func() error) error

Do admits a request if there is room and runs fn, observing fn's outcome; when the limit is reached it returns ErrOverloaded without running fn. The slot is released even if fn panics.

resp, err := do(...)
if errors.Is(err, ErrOverloaded) { http.Error(w, "overloaded", 429); return }

As with Limiter.Do, Config.Classify decides what a failure tells the controller. By default a failed handler is still observed as an ordinary handling time; classify the "failed without doing any work" errors as Unreachable so they cannot pass for a fast, healthy handler.

func (*ConcurrencyLimiter) InFlight

func (l *ConcurrencyLimiter) InFlight() int

InFlight returns the number of requests currently being served.

func (*ConcurrencyLimiter) Limit

func (l *ConcurrencyLimiter) Limit() int

Limit returns the current concurrency limit, in requests.

func (*ConcurrencyLimiter) Observe

func (l *ConcurrencyLimiter) Observe(d time.Duration)

Observe records the handling time of one admitted request. Feed it the handler's own duration, measured from the moment it was admitted to when it finished. Because admission never blocks, this is clean by construction — there is no throttle wait to accidentally contaminate the signal. Safe for concurrent use and allocation-free.

A normal sample also clears probe mode, exactly as Limiter.Observe does: the first request that completes proves the service is answering again, so the clamp must return to FloorLimit. Without this the limit stayed at ProbeLimit for the life of the process after a single Unavailable.

func (*ConcurrencyLimiter) ObserveOutcome

func (l *ConcurrencyLimiter) ObserveOutcome(d time.Duration, err error)

ObserveOutcome records how one request ended, applying Config.Classify to the error. Use it from an integration that admits, times and classifies requests itself, so the error policy stays in one place. It is exactly what Do does with the same two values.

func (*ConcurrencyLimiter) ObserveTruncated

func (l *ConcurrencyLimiter) ObserveTruncated(d time.Duration)

ObserveTruncated records a request whose client went away after d, before the handler finished — see Limiter.ObserveTruncated. Its handling time is at least d, so it is recorded only when d is already past SLOTarget.

Inbound this matters more than outbound: a client's timeout reaches the server as a cancellation, not as a slow answer. When the service is slow enough that clients give up, the requests they give up on are all of the slow ones, and dropping them left the limiter with no samples at all — it sat at whatever limit it had while every request timed out.

func (*ConcurrencyLimiter) Release

func (l *ConcurrencyLimiter) Release()

Release frees one slot taken by TryTake.

func (*ConcurrencyLimiter) Start

func (l *ConcurrencyLimiter) Start()

Start launches the control loop in the background. Calling it more than once is a no-op: a second loop would double the step rate, and since every gain in the control law is a per-tick increment, that silently doubles them too.

func (*ConcurrencyLimiter) Stats

func (l *ConcurrencyLimiter) Stats() Stats

Stats returns a snapshot of the controller's internal signals. For a concurrency limiter, the Rate field holds the current limit, not a rate.

func (*ConcurrencyLimiter) Stop

func (l *ConcurrencyLimiter) Stop()

Stop halts the control loop. Safe to call more than once.

func (*ConcurrencyLimiter) TryTake

func (l *ConcurrencyLimiter) TryTake() bool

TryTake admits the caller if there is room under the current limit and reports whether it was admitted. It never blocks: at the limit it returns false at once, so an inbound caller sheds load instead of queuing. Every true result must be paired with exactly one Release.

func (*ConcurrencyLimiter) Unavailable

func (l *ConcurrencyLimiter) Unavailable()

Unavailable records one request that failed without the service doing any work — a dependency is down and the handler answers 503 immediately, a circuit breaker is open, a request is rejected before touching the hot path. Record it instead of Observe for that request: its handling time is ~0 and would read as a fast, healthy service, so the controller would raise the limit while the service is struggling. A tick in which these requests outnumber the observed ones two to one backs the limit off one step toward ProbeLimit; a tick with fewer only lets the floor fall to ProbeLimit until the next observed request (see Limiter.Unavailable).

An outage tick backs off only once the controller has something to anchor a synthetic sample against: with no SLOTarget and no samples yet there is no honest number to record, and inventing one would corrupt the baseline.

Use this from the manual TryTake/Observe path, or return Unreachable from Config.Classify to have Do call it for you. Never do both for the same request.

type Config

type Config struct {

	// Preset fills the *policy* fields you leave at zero: the anchor, the tick
	// hysteresis, the tolerated multiple, the tail tolerance. Supply MaxRate and
	// SLOTarget and nothing else to get the recommended limiter.
	//
	// Zero (PresetNone) is the recommendation: the SLO anchor with a 5% tail
	// tolerance when SLOTarget is set, the relative law when it is not. Name a
	// preset to pin a law instead. Any field you set explicitly wins over the
	// preset; see applyPreset for the exact table.
	Preset Preset

	// MinRate is the legacy fallback for FloorRate and ProbeRate. New code
	// should set the two explicitly.
	//
	// FloorRate is the normal operating floor: while the downstream is healthy
	// the controller will not go below it. Set it near the static rate you
	// would trust in the worst healthy case; the adaptive part is then upside,
	// not a bet on finding the knee from a low floor.
	//
	// ProbeRate is the outage floor: when a call never reached the downstream,
	// the controller is allowed to fall this low to keep probing without
	// hammering a dependency that is already down. It must be <= FloorRate.
	//
	// MaxRate is the hard ceiling. MinRate is kept as a compatibility fallback:
	// if FloorRate/ProbeRate are zero they inherit MinRate.
	MinRate   int
	FloorRate int
	ProbeRate int
	MaxRate   int
	// InitialRate is the starting rate. When zero it is derived from MaxRate
	// (MaxRate/8, at least FloorRate): the controller comes down fast and goes
	// up slowly, so guessing high costs a couple of ticks while guessing low
	// costs ln(MaxRate/FloorRate) of them.
	InitialRate int

	// SLOTarget enables the tail-latency guard: when the per-tick p99 exceeds
	// it, the controller forces a full backoff step. Zero disables the guard.
	//
	// It plays two other roles: it caps the relative window from above (the
	// controller never aims past sloWindowFraction x SLOTarget), and under
	// AnchorSLO it *is* the window. The guard itself stays ungated by
	// BackoffTicks: a tail that blows through the SLO is a safety valve, and
	// delaying it one tick is not worth the noise it would filter.
	SLOTarget time.Duration

	// TailFactor scales SLOTarget for the tail guard only: the guard fires when
	// the per-tick p99 exceeds SLOTarget x TailFactor. The window cap
	// (0.8 x SLOTarget) and the AnchorSLO target are unaffected.
	//
	// 1.0 (the default) is the historical behaviour: the guard fires the moment
	// the p99 crosses the SLO. That is too strict for most services. A per-tick
	// p99 crossing a number you also promise as an SLO is *normal* when you meet
	// that SLO, and the guard takes -30% on every tick it fires, so firing on
	// half the ticks is enough to walk the rate to MinRate. Measured on a
	// downstream with no queueing at all: 5% of requests that always cost 300ms
	// took the rate from 300 to 21 req/s.
	//
	// 1.5-2.0 is the recommended range when the tail is made of different request
	// types (a report endpoint, a large payload, a cold shard) rather than of
	// queueing. It is a separate knob from SLOTarget because raising that would
	// also widen the window cap and move the SLO anchor.
	TailFactor float64

	// TailTolerance switches the tail guard from a quantile test to a ratio test:
	// it fires when the smoothed fraction of requests above the tail target
	// exceeds this number. Zero leaves it to the Preset: PresetNone with an
	// SLOTarget uses 0.05, and every named preset keeps the quantile test, "is
	// the per-tick p99 above the target".
	//
	// The ratio test exists because a quantile cannot tell a heterogeneous
	// population from queueing: 5% of requests that are simply a different
	// endpoint put the p99 permanently above the target, and the guard then takes
	// -30% per tick until the rate is at MinRate — measured on a downstream with no
	// queueing at all, 300 -> 21 req/s. A ratio says "5% was slow", which against
	// a 10% tolerance is not congestion; when the whole tick is slow the ratio is
	// 100% and the guard fires as before.
	//
	// 0.05-0.10 is the useful range: it is the share of traffic you are willing to
	// see above the target. The ratio is smoothed over about two ticks, so one slow
	// request in a quiet tick decays instead of punishing the rate.
	TailTolerance float64

	// Anchor chooses what the backoff threshold is measured against: the
	// downstream's own fastest latency (the default) or the absolute SLO. See
	// Anchor for what each one buys and costs.
	Anchor Anchor

	// BackoffTicks and RecoverTicks are the time hysteresis: the rate only moves
	// after that many *consecutive* ticks agree about which side of the window
	// the latency is on. This is CoDel's interval idea reduced to a counter, and
	// it is the cheapest defence against a noisy signal: with the asymmetric
	// steps (-DownStep vs +UpStep) a single-tick spike and a single-tick dip do
	// not cancel out — they ratchet the rate down — so filtering them by
	// agreement matters more than the size of the steps.
	//
	// A tick in the deadband resets both counters: the two sides must agree in a
	// row, not merely add up.
	//
	// Zero takes the default: 1 (no hysteresis, the historical behaviour) under
	// AnchorBaseline, and 2 under AnchorSLO, where the threshold is fixed and
	// jitter across it is therefore the main false-positive risk.
	BackoffTicks int
	RecoverTicks int

	// SlowHalfLife is the healthy-baseline EWMA half-life. SlowDownHalfLife
	// controls how quickly it follows improvements down.
	SlowHalfLife     time.Duration // healthy baseline (creeps up slowly)
	SlowDownHalfLife time.Duration // baseline follows improvements down

	// MaxBaselineRise caps how fast the healthy baseline may creep upward, as a
	// fraction per tick. It is the hard limit on "baseline pollution": without
	// it a sustained overload drags the baseline up toward the congested
	// latency, the target window rises with it, and the controller freezes in
	// the overloaded state — congestion widens its own tolerance. Keep it small
	// (well under 1% per tick); a genuinely permanent slowdown is then absorbed
	// over minutes rather than seconds. Zero takes the default.
	MaxBaselineRise float64

	// Tick is the control-loop period.
	Tick time.Duration

	// TargetInflation is how far above the healthy baseline latency the
	// controller lets this tick's p50 rise before it backs off. 1.0 means
	// "accept up to 2x the baseline". Setting it too low makes the limiter back
	// off at the first sign of queuing and run at very low utilization.
	TargetInflation float64
	// UpStep / DownStep cap the fractional change per tick (asymmetry: down
	// faster than up). E.g. 0.05 up, 0.30 down.
	UpStep   float64
	DownStep float64

	// SlowStartGain is the multiplicative increase used while the controller has
	// never yet had to back off — i.e. while it does not know where the knee is.
	// The default 2.0 doubles each tick, so reaching MaxRate costs
	// log2(MaxRate/InitialRate) ticks instead of ln(MaxRate/MinRate)/ln(1+UpStep).
	// With 10 -> 500 and a 1s tick that is 6 ticks instead of 80. This is TCP's
	// slow start for the same reason: the asymmetry that makes additive increase
	// safe — backing off is 6x faster than probing up — is also what makes
	// starting slow unnecessarily expensive.
	//
	// Overshoot costs one or two ticks (DownStep is 0.30), so the risk is
	// bounded; set 1 to get pure AIMD from the first tick.
	SlowStartGain float64
	// Deadband is the hold band around the target latency, as a fraction of it.
	Deadband float64

	// Classify maps a failed call to what the controller is told about it. It is
	// only consulted for non-nil errors; a nil error always means Reached.
	//
	// The default (nil) treats every error as Reached, which is right for a
	// downstream that answered you — a 5xx or a timeout is a real round trip and
	// a real congestion signal. It is wrong for a downstream that never answered:
	// a refused connection returns in a few microseconds, the controller reads
	// that as "the downstream is fast and healthy", and it raises the rate into a
	// dependency that is already down. Return Unreachable for those. Return
	// Aborted for caller-side cancellations, which say nothing about the
	// downstream at all.
	Classify func(err error) Outcome

	// Clock defaults to the real clock. Inject a fake one in tests.
	Clock Clock
}

Config configures a Limiter. Zero fields take sensible defaults (see withDefaults).

The fields are grouped by who should decide them. The first group is the whole configuration for most services; the rest are escape hatches, and each banner says what it costs to move them.

func (Config) Validate

func (c Config) Validate() error

Validate reports the configurations that New would otherwise repair silently, and the ones it cannot make sense of at all.

It validates the config you *wrote*, not the resolved one: every zero field means "use the default", which is always valid. Call it before New if you want a bad config to fail loudly — New keeps its signature, because a library that starts returning errors on upgrade breaks every caller at once.

The repairs it catches are worth knowing by name, because each one used to happen in silence: MinRate above MaxRate was clamped, Deadband of 1 turned every latency into "healthy", and an AnchorSLO without SLOTarget quietly ran the relative law.

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter is an adaptive, downstream-protecting rate limiter. Call Take before each outbound request and Observe the downstream round-trip time afterwards.

func New

func New(cfg Config) *Limiter

New builds a Limiter. It does not start the control loop; call Start (or drive ctrl.step yourself, as tests do).

func (*Limiter) Config

func (l *Limiter) Config() Resolved

Config returns the configuration this limiter is actually running with.

func (*Limiter) Do

func (l *Limiter) Do(fn func() error) error

Do runs fn as one rate-limited downstream call: it takes a slot from the bucket, times fn, and feeds the outcome back to the controller. Callers never touch Take/Observe directly, so they cannot accidentally feed the wrong latency — the clock starts the moment the slot is granted, which is exactly the downstream RTT the controller wants.

fn must be the downstream call itself, performed synchronously and without internal retries: the observed value is one attempt's round trip. If fn sends asynchronously and returns before the response arrives, Do cannot measure the RTT — use Take/Observe by hand in that case.

How a failure is reported to the controller is decided by Config.Classify. By default an error is still observed as an ordinary round trip, which is right for a downstream that answered (a timeout, a 5xx); classify the "never reached" errors as Unreachable so a refused connection cannot masquerade as a fast healthy response.

func (*Limiter) Observe

func (l *Limiter) Observe(rtt time.Duration)

Observe records one downstream round-trip time. Feed it the latency measured from the moment the request was actually sent, NOT including this limiter's own throttle wait. Safe for concurrent use and allocation-free.

func (*Limiter) ObserveOutcome

func (l *Limiter) ObserveOutcome(d time.Duration, err error)

ObserveOutcome records how one call ended, applying Config.Classify to the error. Use it from an integration that measures and classifies calls itself — a http.RoundTripper, a gRPC interceptor — so the error policy stays in one place instead of being reimplemented at every seam. It is exactly what Do does with the same two values.

func (*Limiter) ObserveTruncated

func (l *Limiter) ObserveTruncated(d time.Duration)

ObserveTruncated records a call that was cut short after d — canceled by the caller before the answer arrived. Its real duration is unknown but at least d. Once d is already past SLOTarget that lower bound is evidence the downstream is slow, however the call would have ended, and it is recorded as d. A shorter one says nothing — the caller may simply have left — and is dropped, as is every one when there is no SLOTarget to compare against.

Dropping them all is what blinds a limiter in the overload it exists for: the calls a caller gives up on are the slowest ones, so the controller is left with no samples, or only the lucky fast ones.

It never ends probe mode: a call nobody waited for does not show that the downstream is answering again.

func (*Limiter) Rate

func (l *Limiter) Rate() float64

Rate returns the current target rate in requests per second.

func (*Limiter) Start

func (l *Limiter) Start()

Start launches the control loop in the background. Calling it more than once is a no-op: a second loop would double the step rate, and since every gain in the control law is a per-tick increment, that silently doubles them too.

func (*Limiter) Stats

func (l *Limiter) Stats() Stats

Stats returns a snapshot of the controller's internal signals.

func (*Limiter) Stop

func (l *Limiter) Stop()

Stop halts the control loop. Safe to call more than once.

func (*Limiter) Take

func (l *Limiter) Take() time.Time

Take blocks until this caller is allowed to send, and returns the planned send time. Use it to build a coordinated-omission-safe latency metric.

func (*Limiter) Unavailable

func (l *Limiter) Unavailable()

Unavailable records one call that never reached the downstream — a refused connection, a DNS failure, an already-open circuit breaker. Record it instead of Observe for that attempt: its round trip is ~0 and would read as a healthy response, so the controller would raise the rate into a dependency that is already down.

A tick in which these calls outnumber the observed ones two to one is an outage: the rate backs off one step toward ProbeRate, the floor that keeps probing so recovery is still detected. A tick with fewer is not — a refused call among many answered ones says the downstream is up — and only lets the floor fall to ProbeRate until the next observed call.

An outage tick backs off only once the controller has something to anchor a synthetic sample against: with no SLOTarget and no samples yet, there is no honest number to record, and inventing one would corrupt the baseline.

Use this from the manual Take/Observe path, or return Unreachable from Config.Classify to have Do call it for you. Never do both for the same attempt: that counts one call twice.

type Outcome

type Outcome int

Outcome says what the controller should be told about a call's result. It exists because "did this call fail?" is not the question that matters — the question is "did it reach the downstream?".

const (
	// Reached means the call reached the downstream and came back, whether it
	// came back fast (2xx), slow (timeout) or unhappy (5xx). Its measured round
	// trip is a genuine congestion signal and is recorded as usual.
	Reached Outcome = iota

	// Unreachable means the call never reached the downstream: connection
	// refused, DNS failure, TLS handshake rejected, an already-open circuit
	// breaker. Its round trip is ~0 and says nothing about the downstream's
	// health, so it is not recorded; instead a tick where such calls outnumber
	// the reached ones two to one counts as an outage and backs off toward
	// ProbeRate — the rate it keeps probing with. See Limiter.Unavailable.
	Unreachable

	// Aborted means the caller gave up, or the call never started. It is not a
	// downstream signal, so the controller is told nothing at all.
	Aborted
)

type Preset

type Preset int

Preset bundles the knobs that are *policy* rather than fact, so a caller only has to supply the two things only they can know: how much they are willing to promise (MaxRate) and how slow is still acceptable (SLOTarget).

A preset fills only the fields the caller left at their zero value, so any explicit setting still wins. The named presets are fixed definitions a service can pin; PresetNone is the library's recommendation and follows it.

const (
	// PresetNone is the zero value: the recommended law.
	//
	// With an SLOTarget that is the SLO anchor (0.8 x SLOTarget, two agreeing
	// ticks per step) with a 5% tail tolerance — PresetLatencyFirst with the
	// ratio guard. It is what held up best on traffic that looks like
	// production: endpoints of different cost, a heavy tail, fast answers that
	// slipped past the verdict. The baseline anchor read the cheap endpoint's
	// share drifting across one half as congestion, the per-tick p99 fired on
	// ordinary pauses, and a burst of fast samples pinned it to the floor for
	// most of an hour; the SLO anchor has no baseline to drag, and the ratio
	// guard tells a slow cohort from a slow tick.
	//
	// Without an SLOTarget there is nothing to anchor to, and it is the relative
	// law, exactly as PresetBalanced.
	PresetNone Preset = iota

	// PresetBalanced is the relative law: the baseline anchor at 2x, single-tick
	// decisions, the per-tick p99 guard, every gain at its default. It was the
	// default before PresetNone moved to the SLO anchor; pin it to keep that
	// behaviour, or to run the relative law with an SLO.
	PresetBalanced

	// PresetLatencyFirst trades utilization for the SLO: the threshold becomes
	// the SLO itself (0.8 x SLOTarget) instead of a multiple of the baseline, and
	// a step needs two consecutive ticks. Requires SLOTarget. Use it when the
	// SLO is a contract and the downstream degrades gracefully until it — see
	// AnchorSLO for the trade-off.
	PresetLatencyFirst

	// PresetThroughputFirst tolerates more queueing (3.5x baseline) and also
	// needs two consecutive ticks, which is the shape that came out of the
	// scenario matrix best on a downstream with a knee. Set MaxRate to the knee:
	// this preset will happily run at the ceiling.
	PresetThroughputFirst
)

func (Preset) String

func (p Preset) String() string

type Resolved

type Resolved struct {
	// Kind is "rate" for a Limiter and "concurrency" for a ConcurrencyLimiter.
	// For the latter the three *Rate fields below hold the concurrency bounds.
	Kind   string
	Preset string

	MaxRate     int
	MinRate     int
	FloorRate   int
	ProbeRate   int
	InitialRate int

	// Anchor is "baseline", "slo", or "baseline (AnchorSLO requested without
	// SLOTarget)". The last form is a config that asked for the absolute anchor
	// and is running the relative one.
	Anchor string
	// Window describes the tolerated band. Only its static part can be known
	// before the first sample — the baseline it is measured against does not
	// exist yet — so it is a formula, not a number.
	Window    string
	SLOTarget time.Duration
	// Tail describes when the p99 guard fires: "200ms (2.0x SLOTarget)", or
	// "off (SLOTarget unset)".
	Tail       string
	TailFactor float64
	// TailTolerance is the ratio-mode threshold; zero means the guard is in
	// quantile mode (the historical behaviour).
	TailTolerance float64

	Tick             time.Duration
	SlowHalfLife     time.Duration
	SlowDownHalfLife time.Duration
	MaxBaselineRise  float64

	TargetInflation float64
	UpStep          float64
	DownStep        float64
	SlowStartGain   float64
	Deadband        float64

	BackoffTicks int
	RecoverTicks int
}

Resolved is a printable snapshot of the configuration a limiter is actually running with: presets applied, defaults filled in, and an SLO anchor that could not be honoured reported instead of hidden.

Config itself cannot be logged usefully — it carries two functions — and "what did my config turn into" is the first question of every debugging session. An adaptive limiter you cannot see into is one you cannot debug.

type Stats

type Stats struct {
	Rate     float64       // current target rate, requests per second
	Baseline time.Duration // slow EWMA playing the role of the healthy baseline
	Fast     time.Duration // this tick's p50, for observability

	// What the last completed tick observed. Samples is the number of
	// observations that tick saw; P50 and P99 are that tick's quantiles, and are
	// zero when Samples is zero.
	//
	// Samples is the signal that the controller is *blind*, not stable: with no
	// samples a step changes nothing, so a frozen rate looks exactly like a
	// steady one. It goes to zero under real conditions — low traffic, or a
	// pipeline that deliberately stops observing (cache hits, for instance). Alert
	// on Samples == 0 for more than a few ticks.
	Samples uint64
	P50     time.Duration
	P99     time.Duration

	// TailRatio is the smoothed fraction of this tick's requests that were above
	// the tail target. It is the number the ratio-mode guard compares against
	// TailTolerance, and it stays zero unless TailTolerance is set.
	//
	// Export it: a non-zero TailRatio with a rate that is not moving tells you the
	// guard is at its threshold and the population, not the load, is deciding.
	TailRatio float64
	// TailThreshold is what TailRatio is compared against: TailTolerance plus
	// tailSigma standard errors, recomputed every tick from the sample count. It
	// is zero in quantile mode.
	//
	// Export it because it turns "why did the guard not fire?" into arithmetic: a
	// low-traffic service has a small n, therefore a wide margin, therefore a
	// threshold far above its tolerance.
	TailThreshold float64

	// ColdStart is true until the controller has had to back off once (or has
	// saturated at MaxRate). While it is true the controller doubles each healthy
	// tick instead of adding UpStep: it is still looking for the knee. The flag
	// never resets for the life of the limiter, so a recovery from an outage is
	// an ordinary additive ramp, not a second slow start.
	ColdStart bool

	// Used is how much of the knob the last tick used: callers that asked
	// Take for a slot, per second, on a Limiter; the most requests in flight
	// at once on a ConcurrencyLimiter. It is zero when nothing went through
	// Take or TryTake.
	//
	// While Used is under half of Rate a healthy tick holds the knob instead of
	// raising it. Export it next to Rate: the gap between them is headroom the
	// limiter has actually seen traffic use, at most a factor of two.
	Used float64
}

Stats is a point-in-time snapshot of the controller's internals, for observability. Track these over time; an adaptive limiter you cannot see into is one you cannot debug.