README
¶
metronome
A protocol-agnostic Go load kernel: drive a unit of work at a controlled, live-adjustable rate across N workers, and measure latency and errors.
metronome knows nothing about HTTP, gRPC, Prometheus, configuration files or UI.
You supply a Runner — one unit of work — and a RateController, and it gives
you a stream of Results and an aggregator that turns them into percentiles.
go get github.com/RomanAgaltsev/metronome
Example
runner := metronome.RunnerFunc(func(ctx context.Context) metronome.Result {
start := time.Now()
resp, err := http.Get("http://localhost:8080/api/users/42")
if err != nil {
return metronome.Result{Start: start, Latency: time.Since(start), Err: err}
}
defer resp.Body.Close()
n, _ := io.Copy(io.Discard, resp.Body)
return metronome.Result{
Start: start,
Latency: time.Since(start),
Code: strconv.Itoa(resp.StatusCode),
Bytes: n,
}
})
d := metronome.Driver{
Runner: runner,
Rate: metronome.Ramp{Start: 10, End: 200, Over: 30 * time.Second},
Workers: 16,
MaxRequests: 5000,
}
stats := metronome.NewStats()
for r := range d.Run(context.Background()) {
stats.Record(r)
}
fmt.Println(stats.Snapshot())
// 5000 requests, 198.7 rps, 0.14% err (0 saturated), p50/p95/p99 12ms/41ms/88ms,
// corrected p95/p99 41ms/89ms, behind schedule 1.2ms
Snapshot is a plain struct — reach for the fields when you want them
(snap.P95, snap.Codes, snap.Bytes). String() is there so every consumer
does not reinvent the same summary line, and so the numbers that matter appear in
the order they should be read.
You must drain the channel until it closes, or cancel the context — abandoning a live channel leaks the workers.
What's in it
| Piece | What it does |
|---|---|
Runner / RunnerFunc |
one unit of work; the integration seam |
Mix(...Weighted) |
weighted random pick among sub-Runners |
Constant, Ramp, Phased |
static rate profiles |
Adaptive + SetRate |
rate driven live from outside (a control loop) |
Driver |
the paced worker pool; Run(ctx) <-chan Result |
Pacing + ClosedLoop / OpenLoop |
how the Driver reacts when the target cannot keep up |
Burst |
rate-limiter burst size; 0 means 1 (smoothest schedule) |
ErrSaturated |
open-loop marker: no worker was free at the scheduled time |
Stats / Snapshot |
HDR-histogram percentiles, error rate, achieved rps, bytes/codes |
Snapshot.Saturated |
how much of the error rate was the target refusing work |
Snapshot.MaxScheduleLag |
how far the generator itself fell behind its own schedule |
Snapshot.String() |
the numbers a run is judged on, in the order to read them |
Snapshot.Corrected* |
coordinated-omission-corrected percentiles (see the caveat below) |
Clock / ManualClock |
injected time, for deterministic tests |
Pacing model — read this before trusting the numbers
metronome offers two pacing modes and reports latency two ways. The defaults are conservative; the honest defaults for a latency-measuring tool are the other ones.
Closed loop (default)
Each worker waits for a rate-limiter token, calls your Runner, and only then asks
for the next token. Simple, self-throttling, and the right choice when you are the
backpressure mechanism (a control loop that deliberately probes for the point where
the target sags). Two consequences:
- Rate sag. When the target slows, workers sit inside
Do, tokens go unclaimed, and the achieved rate falls below the offered rate — silently. - Coordinated omission. Latency is measured from when work actually started, so every stall suppresses exactly the samples that would have revealed it.
Open loop (Pacing: metronome.OpenLoop)
One dispatcher keeps the schedule and never blocks on the target. A unit that finds
no free worker is delivered immediately as a Result whose Err matches
errors.Is(err, metronome.ErrSaturated). Saturation therefore shows up as
Snapshot.ErrorRate, not as invisible sag — and Snapshot.Saturated counts it
separately, so "my generator ran out of workers" stays distinguishable from "the
target failed". Workers becomes the maximum in-flight cap.
Open loop paces from one dispatcher goroutine, which sleeps once per unit of
work. Below roughly a 250µs interval that sleep costs more than the interval, and the
dispatcher — not the target — becomes the bottleneck: on the machine measured below,
open loop holds 1,000 rps exactly and delivers only 74% of 5,000 rps, with zero
ErrSaturated results, because the target was never asked. Snapshot.MaxScheduleLag
is what makes that visible; see the table below. Closed loop does not have this ceiling
(its Workers goroutines sleep concurrently, so each individual sleep is Workers
times longer). Above ~1,000 rps, check MaxScheduleLag, and prefer closed loop if
throughput matters more to you than schedule fidelity.
Raw vs corrected percentiles
Every Result from a Driver carries Scheduled, the time it should have been
sent. Stats reports both:
P50/P95/P99— measured from when work started. Optimistic under stall.CorrectedP50/CorrectedP95/CorrectedP99— measured from when the work was due, i.e. including the queueing delay a schedule-faithful client would have suffered.
Read them as a pair. A large gap means the generator queued: the raw numbers understate what a real client would experience by roughly that amount.
snap := stats.Snapshot()
fmt.Printf("p95 %v (corrected %v), achieved %.1f rps, %v behind schedule\n",
snap.P95, snap.CorrectedP95, snap.RPS, snap.MaxScheduleLag)
The schedule is anchored: Scheduled is the run's origin plus one interval per
unit dispatched, advanced whether or not the generator got there on time. A late
generator is therefore measured against the schedule rather than redefining it. One
worker offered 100 rps against a 50ms target achieves 19.9 rps and reports:
raw P50=50.0ms P99=50.0ms
corrected P50=412.9ms P99=816.1ms MaxScheduleLag=765.7ms
The 816ms is the honest number: by the twentieth request, a client that kept to the 10ms schedule would have been waiting that long. (Before v0.3 both lines read 50ms.)
Who fell behind — you or the target?
Snapshot.Saturated and Snapshot.MaxScheduleLag answer that between them:
Saturated |
MaxScheduleLag |
Meaning |
|---|---|---|
| 0 | ~0 | the schedule was kept and the target kept up |
| > 0 | ~0 | the target could not keep up — open loop working as designed |
| 0 | large | the generator could not keep up — lower the rate, or use closed loop |
The third row is why MaxScheduleLag exists: ErrSaturated can only report a target
that was too slow, never a dispatcher that was. At 5,000 rps open loop reports
Saturated=0, MaxScheduleLag=330ms — nothing was wrong with the target.
Measured accuracy
Measured on an AMD Ryzen 5 3600 (6 cores / 12 threads), Windows 11, Go 1.26.6, machine otherwise idle. Adherence is achieved rps ÷ offered rps; 1.00 is perfect.
| Offered rate | Closed-loop adherence | Open-loop adherence |
|---|---|---|
| 10 rps | 1.00 | 1.00 |
| 100 rps | 1.00 | 1.00 |
| 1,000 rps | 1.00 | 1.00 |
| 5,000 rps | 1.00 | 0.74 |
The 0.74 is real and reproducible, and it is a limit of the generator, not of the
target — see the open-loop note above. That run reports Saturated=0 and
MaxScheduleLag=330ms, which is how you would tell without reading this table.
Anchoring the schedule in v0.3 made the shortfall visible, not smaller: the
adherence numbers are unchanged from v0.2. Reproduce with:
go test -run '^$' -bench BenchmarkDriverPaced -benchtime 3x ./...
Per-request kernel overhead (no-op Runner, unlimited rate, Workers = GOMAXPROCS):
600 ns/op closed-loop and 895 ns/op open-loop, i.e. a plumbing ceiling near
1.7M rps and 1.1M rps respectively. Note the gap between that ceiling and the
5,000 rps adherence figure above: metronome's limit at realistic rates is sleep
granularity, not CPU. Stats.Record costs 211 ns/op under full contention.
Reproduce with:
go test -run '^$' -bench 'BenchmarkDriverOverhead|BenchmarkStatsRecord' ./...
Status
v0.3 — API is stable in shape and pinned by two consumers, but pre-v1: minor versions may carry small breaking changes, always with a migration note in the CHANGELOG. Pin an exact version.
Used by
- crescendo — PromQL-driven adaptive load: a
Mixof endpoint Runners under anAdaptivecontroller steered by a Prometheus feedback loop. - quiver —
qv load: one Runner over a saved API request,Constant/Ramp,MaxRequests,Statssummary.
License
MIT
Documentation
¶
Overview ¶
Package metronome drives a unit of work at a controlled, live-adjustable rate across N workers and measures latency and errors. It is protocol-agnostic: it knows nothing about HTTP, gRPC, Prometheus, configuration, or UI.
The two seams are Runner (what to send) and RateController (how fast). Driver paces the work. Stats aggregates the resulting Result stream into percentile [Snapshot]s.
Pacing model ¶
Driver.Pacing selects between two modes.
ClosedLoop (the default, and v0.1's only behaviour) gives each worker a rate-limiter token, runs the work, and only then asks for the next token. It is self-throttling, but when the target slows the achieved rate sags below the offered rate — silently.
OpenLoop paces from a single dispatcher that never blocks on the target. A unit that finds no free worker is delivered immediately as a Result whose Err matches ErrSaturated, so saturation is counted rather than absorbed. Workers becomes the maximum in-flight cap. The dispatcher sleeps once per unit, so above roughly 4,000 rps it becomes the bottleneck itself; see the README's measured accuracy table.
Compare Snapshot.RPS against the rate you asked for on every run, in either mode.
Raw and corrected percentiles ¶
Every Result produced by a Driver carries Result.Scheduled, the time the unit was due. The schedule is anchored: Scheduled is the run's origin plus one interval per unit dispatched, advanced whether or not the generator got there on time, so a late generator is measured against the schedule rather than redefining it.
Stats reports raw percentiles (P50/P95/P99, from when work started) alongside coordinated-omission-corrected ones (CorrectedP50/P95/P99, from when the work was due). Read them as a pair: a large gap means the generator queued, and the raw numbers understate what a real client would have suffered by roughly that much.
Diagnosing a run that fell short ¶
Snapshot.Saturated counts units the target had no free worker for; Snapshot.MaxScheduleLag is how far the generator itself fell behind. Saturation with no lag means the target could not keep up. Lag with no saturation means metronome could not — lower the rate, or use ClosedLoop.
Index ¶
Examples ¶
Constants ¶
const DefaultResultBuffer = 1024
DefaultResultBuffer is the capacity of Run's result channel when Driver.ResultBuffer is zero.
const DefaultWorkers = 10
DefaultWorkers is the worker count Run uses when Driver.Workers is not positive.
Variables ¶
var ErrSaturated = errors.New("metronome: no worker free at scheduled time")
ErrSaturated marks an open-loop unit of work that found no free worker at its scheduled time. It is recorded rather than silently dropped, so saturation appears in Snapshot.ErrorRate instead of as invisible rate sag. Test for it with errors.Is.
Functions ¶
This section is empty.
Types ¶
type Adaptive ¶
type Adaptive struct {
// contains filtered or unexported fields
}
Adaptive is a RateController whose rate is set externally. Safe for concurrent SetRate/Rate.
func NewAdaptive ¶
NewAdaptive returns an Adaptive controller starting at initial rps.
func (*Adaptive) Rate ¶
Rate reports the rate most recently passed to SetRate, ignoring elapsed time.
func (*Adaptive) SetRate ¶
SetRate sets the rate reported from the next Rate call onward. It is safe to call concurrently with Rate and from any goroutine.
A rate of zero or less is floored by the Driver to a token every ~10,000 seconds: effectively paused, and resumable — a later SetRate takes effect within one rate-update interval (see maxReservationWait), so pausing is not a one-way door. NaN is floored the same way rather than treated as "unlimited", because a control loop dividing by an empty PromQL vector produces NaN and flooding the target is the wrong direction to fail; see sanitizeRate.
type Clock ¶
type Clock interface {
// Now reports the current time.
Now() time.Time
// Sleep blocks for d, returning nil, or ctx.Err() if ctx is done first. A
// non-positive d returns immediately with ctx.Err() (nil for a live ctx).
Sleep(ctx context.Context, d time.Duration) error
}
Clock supplies time to the Driver. It is injected so that pacing is testable: with a ManualClock the whole pacing path — reservations, sleeps and the rate update cadence — advances only when the test says so.
func SystemClock ¶
func SystemClock() Clock
SystemClock returns a Clock backed by the wall clock. It is what the Driver uses when Driver.Clock is nil.
type Driver ¶
type Driver struct {
Runner Runner
Rate RateController
Workers int
MaxRequests int
// Clock supplies time to the whole pacing path: the rate schedule, the
// sleeps between units of work, and the rate-update cadence. Leave nil for
// the wall clock; inject a ManualClock to drive pacing exactly in tests.
Clock Clock
// ResultBuffer is the capacity of the channel Run returns. Zero selects
// DefaultResultBuffer; a negative value selects an unbuffered channel.
//
// Buffering matters for measurement, not just throughput: on an unbuffered
// channel a consumer that pauses blocks a worker that is holding a
// rate-limiter token, producing rate sag the target did not cause.
ResultBuffer int
// Pacing selects ClosedLoop (default) or OpenLoop. See PacingMode.
Pacing PacingMode
// Burst is the rate limiter's burst size; 0 means 1. Burst 1 gives the
// smoothest schedule. A larger burst lets the generator catch up in a bunch
// after a stall, which is sometimes what you want and is never smooth.
Burst int
}
Driver runs a Runner under a RateController across Workers goroutines, emitting Results on the channel Run returns until ctx is cancelled or MaxRequests results have been produced (MaxRequests == 0 means unlimited).
Contract: when MaxRequests is set, exactly MaxRequests Results are delivered unless ctx is cancelled first (cancellation aborts promptly and may drop in-flight results). In OpenLoop mode a saturated attempt counts as one of them. Results are NOT ordered — in OpenLoop a saturated unit is emitted immediately while an earlier unit is still running. The channel is closed once all internal goroutines exit; the caller must drain it or cancel ctx.
Example ¶
package main
import (
"context"
"fmt"
"time"
"github.com/RomanAgaltsev/metronome"
)
func main() {
runner := metronome.RunnerFunc(func(context.Context) metronome.Result {
return metronome.Result{Start: time.Now(), Latency: time.Millisecond}
})
d := metronome.Driver{Runner: runner, Rate: metronome.Constant(500), Workers: 4, MaxRequests: 100}
stats := metronome.NewStats()
for r := range d.Run(context.Background()) {
stats.Record(r)
}
fmt.Println(stats.Snapshot().Count)
}
Output: 100
func (*Driver) Run ¶
Run starts the workers and returns the channel Results are delivered on.
Run has a pointer receiver, so it cannot be called on a composite literal: bind the Driver to a variable first (d := Driver{...}; d.Run(ctx)).
The caller MUST drain the returned channel until it closes, or cancel ctx. Abandoning a live channel leaves the workers blocked on the send and leaks them for the lifetime of the process.
Run panics if Runner or Rate is nil — programmer errors, not runtime conditions. Workers <= 0 defaults to DefaultWorkers.
type ManualClock ¶
type ManualClock struct {
// contains filtered or unexported fields
}
ManualClock is a Clock whose time only advances when Advance is called. Goroutines sleeping on it wake when Advance moves the clock to or past their deadline, which makes pacing tests exact instead of tolerance-based.
func NewManualClock ¶
func NewManualClock(t time.Time) *ManualClock
NewManualClock returns a ManualClock reading t.
func (*ManualClock) Advance ¶
func (m *ManualClock) Advance(d time.Duration)
Advance moves the clock forward by d, waking every sleeper whose deadline the clock has now reached, in deadline order.
func (*ManualClock) BlockUntilSleepers ¶ added in v0.2.0
func (m *ManualClock) BlockUntilSleepers(n int)
BlockUntilSleepers blocks until at least n goroutines are asleep on this clock. It exists to remove the race between a goroutine reaching Sleep and a test calling Advance: without it, an Advance that lands first is simply lost and the test hangs.
It has no timeout on purpose — if the count is never reached, go test's own deadline reports it with every goroutine's stack, which is more useful than a bespoke error.
func (*ManualClock) Now ¶
func (m *ManualClock) Now() time.Time
Now reports the clock's current time.
type PacingMode ¶ added in v0.2.0
type PacingMode int
PacingMode selects how the Driver reacts when the target cannot keep up.
const ( // ClosedLoop is the default: a worker does not ask for its next token until // the current unit of work has completed. Simple and self-throttling, but a // slow target reduces the achieved rate, and latency percentiles are subject // to coordinated omission (Stats' Corrected* fields quantify how much). ClosedLoop PacingMode = iota // OpenLoop keeps the schedule regardless of the target: one dispatcher paces // and hands each unit to a free worker, or — if all Workers are busy — // records a Result carrying ErrSaturated and moves on. Saturation becomes a // counted signal instead of silent rate sag. Workers is the maximum // in-flight cap in this mode. OpenLoop )
type PanicError ¶ added in v0.2.0
PanicError reports a panic raised inside a Runner. The Driver recovers it so that one bad unit of work cannot abort a whole load run; the panic is delivered as a failed Result instead, and Stack holds the stack captured at the point of recovery.
func (*PanicError) Error ¶ added in v0.2.0
func (e *PanicError) Error() string
type Phased ¶
type Phased struct {
Phases []Phase
}
Phased steps through phases by elapsed time, holding the last phase's rate past the end.
type RateController ¶
RateController decides the target rate (requests/sec) over elapsed time.
type Result ¶
type Result struct {
// Scheduled is the time this unit of work was due per the pacing schedule —
// the run's origin plus one interval for each unit dispatched before it,
// advanced independently of when the generator actually got to it.
//
// Start minus Scheduled is therefore the generator's own queueing delay: the
// wait a client that never fell behind would have suffered before its
// request even started. It is what Stats uses to correct for coordinated
// omission, and it is aggregated as Snapshot.MaxScheduleLag.
//
// It may be in the past relative to Start (the generator was late — the
// whole point) or in the future when Burst > 1 lets several units go at once
// and the schedule says the later ones were not due yet; Stats floors the
// correction at zero for that case. Zero if the Result did not come from a
// Driver.
Scheduled time.Time
Start time.Time
Latency time.Duration
Err error
Code string // status/gRPC code, caller-supplied
Bytes int64
// Labels is caller-side metadata carried alongside the Result, e.g.
// {"endpoint": "get_user"}. Stats does not aggregate it: a per-label
// breakdown helper is roadmapped, and until it exists a caller who needs
// one keys their own map off these Results. Leave it nil if unused — it is
// an allocation per request.
Labels map[string]string
}
Result is the outcome of one unit of work.
type RunnerFunc ¶
RunnerFunc adapts a function to a Runner.
type Snapshot ¶
type Snapshot struct {
Count int64
Errors int64
RPS float64
ErrorRate float64
P50, P95, P99 time.Duration
Max time.Duration
// Clamped counts Results whose Latency fell outside the histogram's range
// and was clamped to a bound — at most one per Result. Non-zero means P50,
// P95 and P99 understate reality at one end; widen the range with
// NewStatsRange. Max always reports the true maximum regardless.
Clamped int64
// CorrectedClamped is the same count for the corrected histogram, which
// records Latency plus queueing delay and therefore overflows a bound the
// raw latency never reaches. It is counted separately because it says
// something about CorrectedP50/P95/P99 only — a run can have
// CorrectedClamped > 0 with Clamped == 0 and percentiles that are fine.
CorrectedClamped int64
// MaxScheduleLag is the largest gap between when a unit of work was due and
// when it actually started. It is the generator's own lateness: non-zero
// means metronome did not keep the schedule it offered, whatever the target
// did. Read it against Saturated — a large lag with Saturated == 0 says the
// generator, not the target, is the bottleneck.
MaxScheduleLag time.Duration
// Saturated counts Results carrying ErrSaturated: open-loop units that found
// no free worker at their scheduled time. They are included in Errors and
// ErrorRate, and this field is what separates "my generator ran out of
// workers" from "the target failed". Always zero in ClosedLoop.
Saturated int64
// Bytes is the sum of Result.Bytes.
Bytes int64
// Throughput is bytes/sec: RPS multiplied by the mean Bytes per Result, so
// it is the same kind of estimate as RPS rather than a raw total over the
// observed span. (Bytes/span would spend all N samples' bytes over the N-1
// intervals N timestamps actually bound — the bias RPS avoids.)
Throughput float64
// Codes counts Results by Result.Code. Results with an empty Code are
// counted in Count but not here. The map is a copy owned by the caller.
Codes map[string]int64
// CorrectedP* are the coordinated-omission-corrected percentiles: each
// Result's latency plus the time it spent waiting past its Scheduled send
// time. They answer "what would a client that kept to the schedule have
// seen?", which is the honest number when the target stalls. They are zero
// when no Result carried a Scheduled stamp.
//
// Read them against the raw P* above: a large gap means the generator
// queued, so the raw numbers understate what a real client would suffer.
CorrectedP50, CorrectedP95, CorrectedP99 time.Duration
// CorrectedCount is how many Results carried a Scheduled stamp and are
// therefore represented in CorrectedP*.
CorrectedCount int64
}
Snapshot is an aggregated view of many Results over a window.
func (Snapshot) String ¶ added in v0.4.0
String renders the numbers a run is usually judged on, in the order they should be read: what was achieved, what it cost, what a schedule-faithful client would have seen, and whether the generator itself kept up.
It is a summary, not a serialisation — Codes, Bytes, Throughput and the clamping counters are omitted. Reach for the fields directly when you need them.
type Stats ¶
type Stats struct {
// contains filtered or unexported fields
}
Stats aggregates Results into percentile Snapshots. Safe for concurrent Record.
func NewStats ¶
func NewStats() *Stats
NewStats returns Stats recording latencies from 1µs to 60s with 3 significant digits — a sensible default for HTTP and gRPC. Use NewStatsRange when your work is slower or faster than that.
func NewStatsRange ¶ added in v0.2.0
NewStatsRange returns Stats recording latencies in [lo, hi] with sigfigs significant digits. Latencies outside the range are clamped to the nearest bound and counted in Snapshot.Clamped rather than dropped; Snapshot.Max always reports the true maximum.
It panics if lo <= 0, hi <= lo, or sigfigs is outside [1, 5] — programmer errors, and there is no error path on a constructor two consumers call at startup.
Source Files