metrics

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: AGPL-3.0 Imports: 14 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// HistogramScale converts a metric's natural unit into histogram units.
	HistogramScale = 1000

	DefaultHistogramLowest  int64 = 100        // 0.1 ms
	DefaultHistogramHighest int64 = 60_000_000 // 60 s
	DefaultSigFigs                = 2          // ~1% relative error
)

Trend metrics arrive as floating point values in their natural unit — milliseconds for every built-in latency metric. HDR stores integers, so values are scaled before recording and unscaled on the way out.

A scale of 1000 with a lowest value of 100 gives a resolution floor of 0.1ms and a ceiling of 60s, which brackets everything an HTTP load test cares to distinguish.

View Source 
const (
	DefaultMaxSeries   = 5000
	DefaultTrendShards = 4
)

Defaults applied to a zero RecorderConfig.

View Source 
const (
	DefaultResolution = time.Second
	DefaultWindow     = time.Hour
	DefaultLateGrace  = 3 * time.Second

	// DefaultMaxChartedEndpoints bounds the per-endpoint timeline.
	//
	// Endpoint names are already collapsed to low cardinality, so this is
	// generous for any real service; it exists so that a run which defeats
	// that collapsing cannot turn a one-hour window into a memory leak.
	DefaultMaxChartedEndpoints = 40
)

Defaults applied to a zero StoreConfig.

View Source 
const DefaultMaxFailureKinds = 100

DefaultMaxFailureKinds bounds how many distinct kinds of failure a node or a run will track.

Kinds, not occurrences: the count of each is unbounded. A run failing in more than a hundred distinguishable ways has a problem the hundred-and-first row was not going to explain.

Variables

View Source 
var PercentileKeys = []string{"p50", "p90", "p95", "p99"}

PercentileKeys names the reported percentiles, in the same order.

Functions

func CountsLen 

func CountsLen(cfg HistogramConfig) int

CountsLen reports how many buckets a histogram with this configuration has.

Safe for concurrent use. Two callers racing on a cold configuration may both compute the value, which is harmless: it is a pure function of cfg, so they agree, and one simply overwrites the other.

func DecodeHistogram 

func DecodeHistogram(snap *loadwavev1.HistogramSnapshot, cfg HistogramConfig) (*hdr.Histogram, error)

DecodeHistogram rebuilds a histogram from its wire form.

The bucket layout implied by the message is checked against cfg before the counts are installed. Skipping that check would let a node running a different resolution corrupt the merged distribution, or index out of bounds inside the HDR library.

func DecodeKind 

func DecodeKind(k loadwavev1.MetricKind) loadwave.MetricKind

DecodeKind maps the wire enum back onto the SDK's metric kind.

func EncodeHistogram 

func EncodeHistogram(h *hdr.Histogram) *loadwavev1.HistogramSnapshot

EncodeHistogram converts a histogram into its wire form.

func UnscaleValue 

func UnscaleValue(v int64) float64

UnscaleValue converts a histogram unit back to the metric's natural unit.

Types

type Bucket 

type Bucket struct {
	Start     time.Time
	ActiveVUs uint32
	Points    map[PointKey]*Point
	Status    map[string]uint64

	// Endpoints is the per-request-name breakdown for this interval.
	Endpoints map[string]*EndpointPoint
	// contains filtered or unexported fields
}

Bucket is the whole cluster's activity over one resolution interval.

type EndpointPoint 

type EndpointPoint struct {
	Requests uint64  `json:"requests"`
	Failures uint64  `json:"failures"`
	Sum      float64 `json:"-"`
	Observed uint64  `json:"-"`
}

EndpointPoint is one request name's activity within one time bucket.

It carries no histogram, and therefore no percentiles. That is what makes a per-endpoint timeline affordable: a histogram per endpoint per second would cost gigabytes over an hour, where a sum and a count cost thirty-two bytes. Percentiles per endpoint are still available for the whole run, from the cumulative aggregates that Endpoints() merges.

func (EndpointPoint) Avg 

func (p EndpointPoint) Avg() float64

Avg returns the mean duration in milliseconds, or zero if nothing was timed.

func (EndpointPoint) ErrorRate 

func (p EndpointPoint) ErrorRate() float64

ErrorRate returns the share of this endpoint's requests that failed.

type EndpointSummary 

type EndpointSummary struct {
	Name        string             `json:"name"`
	Requests    uint64             `json:"requests"`
	Failures    uint64             `json:"failures"`
	ErrorRate   float64            `json:"errorRate"`
	Avg         float64            `json:"avg"`
	Min         float64            `json:"min"`
	Max         float64            `json:"max"`
	Percentiles map[string]float64 `json:"percentiles,omitempty"`
	Statuses    map[string]uint64  `json:"statuses,omitempty"`
	BytesIn     float64            `json:"bytesIn"`
}

EndpointSummary is one request name's whole-run view, merged across every status code and scenario that used it.

type FailureSummary 

type FailureSummary struct {
	Name       string    `json:"name"`
	Method     string    `json:"method"`
	Status     int32     `json:"status"`
	ErrorClass string    `json:"errorClass,omitempty"`
	Message    string    `json:"message,omitempty"`
	Count      uint64    `json:"count"`
	LastSeen   time.Time `json:"lastSeen"`
}

FailureSummary is one kind of failure over the whole run.

type HistogramConfig 

type HistogramConfig struct {
	// Lowest is the smallest distinguishable value, in histogram units.
	Lowest int64
	// Highest is the largest trackable value, in histogram units. Larger
	// observations are clamped to it and counted as clipped.
	Highest int64
	// SigFigs is the number of significant decimal digits kept, which sets
	// the relative error and, with it, the memory each histogram occupies.
	SigFigs int
}

HistogramConfig fixes the resolution of every trend metric in a run.

Every node must use identical settings: HDR histograms can only be merged when their bucket layouts match, and a mismatch is what turns a distributed p99 into nonsense. The coordinator therefore pins the configuration and nodes never choose their own.

func DefaultHistogramConfig 

func DefaultHistogramConfig() HistogramConfig

DefaultHistogramConfig returns the resolution used unless a run overrides it.

func (HistogramConfig) Equal 

func (c HistogramConfig) Equal(other HistogramConfig) bool

Equal reports whether two configurations produce mergeable histograms.

func (HistogramConfig) New 

func (c HistogramConfig) New() *hdr.Histogram

New allocates a histogram with this configuration.

func (HistogramConfig) ScaleValue 

func (c HistogramConfig) ScaleValue(v float64) int64

ScaleValue converts a metric value in its natural unit into histogram units, clamped into the trackable range.

Clamping rather than dropping is deliberate: an observation above the ceiling is nearly always a genuine, interesting outlier — a request that hit its timeout — and silently discarding it would make the tail look better than it is. Recording it at the ceiling understates it but keeps it counted.

func (HistogramConfig) Validate 

func (c HistogramConfig) Validate() error

Validate reports whether the configuration can build a histogram.

type Point 

type Point struct {
	Count   uint64  `json:"count"`
	Sum     float64 `json:"sum"`
	Min     float64 `json:"min"`
	Max     float64 `json:"max"`
	NonZero uint64  `json:"nonZero"`

	// Percentiles is populated for trend metrics, keyed by PercentileKeys.
	Percentiles map[string]float64 `json:"percentiles,omitempty"`
}

Point is one metric's activity inside one time bucket.

It holds scalars only. The histogram a bucket needs to derive percentiles is discarded once the bucket is finalised: keeping one per series per second for an hour-long run would cost tens of gigabytes, whereas the four percentiles the dashboard actually plots cost thirty-two bytes.

func (Point) Avg 

func (p Point) Avg() float64

Avg returns the mean observation, or zero when there were none.

func (Point) Ratio 

func (p Point) Ratio() float64

Ratio returns the share of truthy observations, for rate metrics.

type PointKey 

type PointKey struct {
	Metric   string `json:"metric"`
	Scenario string `json:"scenario"`
}

PointKey identifies a bucket series.

Buckets are indexed by metric and scenario only, not by the full label set. Full-cardinality history is what makes a metrics store expensive, and the live charts plot exactly these two dimensions; per-label detail is available cumulatively through Summary, which is what the endpoint table renders from.

type Recorder 

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder is a node's metric aggregator: the sink virtual users write to and the source the reporting loop flushes from.

Recording is on the hottest path in the program — several observations per HTTP request, tens of thousands of requests per second — so it does no allocation in steady state and takes only a sharded mutex.

Safe for concurrent use.

func NewRecorder 

func NewRecorder(cfg RecorderConfig) *Recorder

NewRecorder builds an aggregator.

func (*Recorder) Config 

func (r *Recorder) Config() RecorderConfig

Config returns the effective configuration, with defaults resolved.

func (*Recorder) Count 

func (r *Recorder) Count(metric string, labels loadwave.Labels, delta float64)

Count adds to a counter. See ForVU for the per-virtual-user path.

func (*Recorder) Dropped 

func (r *Recorder) Dropped() uint64

Dropped reports how many observations were discarded for exceeding the series cap. A non-zero value means the reported numbers understate reality, which the dashboard surfaces rather than hides.

func (*Recorder) Flush 

func (r *Recorder) Flush(
	runID, nodeID string, bucketStart time.Time, width time.Duration, activeVUs uint32,
) *loadwavev1.MetricBatch

Flush drains everything recorded since the previous call and returns it as a wire batch, leaving the recorder empty and ready for the next interval.

Batches carry deltas rather than running totals. A dropped batch then costs exactly one interval of data instead of skewing every interval that follows, which matters because a node reconnecting after a partition is a routine event rather than an exceptional one.

func (*Recorder) ForVU 

func (r *Recorder) ForVU(vuID int64) loadwave.Recorder

ForVU returns a Recorder view pinned to one shard.

Pinning by virtual user rather than choosing a shard per observation spreads load evenly while keeping each VU's writes on one lock, which is both faster and kinder to the cache than a global round robin.

func (*Recorder) Gauge 

func (r *Recorder) Gauge(metric string, labels loadwave.Labels, value float64)

Gauge sets the current value of a gauge.

func (*Recorder) Rate 

func (r *Recorder) Rate(metric string, labels loadwave.Labels, ok bool)

Rate records one boolean observation.

func (*Recorder) ReportFailure 

func (r *Recorder) ReportFailure(f loadwave.Failure)

ReportFailure implements loadwave.FailureReporter.

Reached only on the failure path, so a healthy run never pays for it. Failures are aggregated by kind, so there is nothing per-virtual-user to keep and the pinned view simply forwards.

func (*Recorder) SeriesCount 

func (r *Recorder) SeriesCount() int

SeriesCount reports how many distinct series have been seen.

func (*Recorder) Trend 

func (r *Recorder) Trend(metric string, labels loadwave.Labels, value float64)

Trend records one observation in a distribution.

type RecorderConfig 

type RecorderConfig struct {
	// Shards is how many independent partitions scalar metrics are spread
	// across, rounded up to a power of two. More shards mean less lock
	// contention between virtual users. Zero picks a value from GOMAXPROCS.
	Shards int

	// TrendShards is the same for trend metrics, and is deliberately smaller.
	// Each shard holds its own histogram for every trend series it sees, and
	// a histogram is kilobytes rather than the few dozen bytes a counter
	// costs — so widening this trades a lot of memory for a little
	// contention. Zero picks a small default. Clamped to at most Shards.
	TrendShards int

	// MaxSeries caps how many distinct series the node will track. Beyond it,
	// observations for new series are dropped and counted rather than
	// allowed to exhaust memory. A runaway tag — a user id used as a label —
	// is the usual cause, and a bounded, visibly lossy run beats an OOM.
	MaxSeries int

	// Histogram fixes trend resolution and must match every other node.
	Histogram HistogramConfig

	// MaxFailureKinds caps how many distinct kinds of failure are tracked.
	// Zero applies DefaultMaxFailureKinds.
	MaxFailureKinds int
}

RecorderConfig tunes the per-node aggregator.

type SeriesSummary 

type SeriesSummary struct {
	Metric      string             `json:"metric"`
	Kind        string             `json:"kind"`
	Tags        map[string]string  `json:"tags,omitempty"`
	Count       uint64             `json:"count"`
	Sum         float64            `json:"sum"`
	Min         float64            `json:"min"`
	Max         float64            `json:"max"`
	Avg         float64            `json:"avg"`
	Rate        float64            `json:"rate"`
	Percentiles map[string]float64 `json:"percentiles,omitempty"`
}

SeriesSummary is one series' cumulative state over the whole run.

type Stats 

type Stats struct {
	Series           int       `json:"series"`
	ClosedBuckets    int       `json:"closedBuckets"`
	OpenBuckets      int       `json:"openBuckets"`
	DroppedSeries    uint64    `json:"droppedSeries"`
	DroppedLate      uint64    `json:"droppedLate"`
	DroppedByNode    uint64    `json:"droppedByNode"`
	DroppedEndpoints uint64    `json:"droppedEndpoints"`
	DroppedFailures  uint64    `json:"droppedFailureKinds"`
	Started          time.Time `json:"started"`
	LastSeen         time.Time `json:"lastSeen"`
}

Stats reports the store's own health, so the dashboard can warn when the numbers it is showing are incomplete.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store accumulates every node's reports into one coherent view of a run.

It keeps two things: a cumulative aggregate per series, at full label cardinality and full histogram fidelity, which drives the endpoint table and threshold evaluation; and a rolling window of time buckets at reduced dimensionality, which drives the live charts.

Safe for concurrent use.

func NewStore 

func NewStore(cfg StoreConfig) *Store

NewStore builds an empty store for one run.

func (*Store) Aggregate 

func (s *Store) Aggregate(metric string) (SeriesSummary, bool)

Aggregate returns the cumulative aggregate for one metric across every label combination, which is what thresholds are evaluated against.

func (*Store) CloseStale 

func (s *Store) CloseStale(now time.Time)

CloseStale finalises every open bucket whose grace period has elapsed.

The coordinator calls this on a ticker rather than only on ingest, so that a run whose traffic stops still gets its final buckets published instead of leaving the chart hanging one interval short.

func (*Store) Endpoints 

func (s *Store) Endpoints() []EndpointSummary

Endpoints returns one row per request name, sorted slowest first by p95.

Series are stored split by status code as well as by name, so a per-endpoint percentile has to be recomputed from the merged distribution of all of that endpoint's slices. Taking the maximum of the per-status percentiles — the obvious shortcut — reports the tail of whichever status happened to be slowest rather than the endpoint's actual tail, and the two diverge most exactly when an endpoint starts failing, which is when the number matters.

func (*Store) Failures 

func (s *Store) Failures() []FailureSummary

Failures returns every kind of failure seen in the run, most frequent first.

func (*Store) Ingest 

func (s *Store) Ingest(batch *loadwavev1.MetricBatch) error

Ingest folds one node's batch into the store.

An error means the batch was rejected outright — a resolution mismatch, or a bucket so old it has already been evicted. Individual series that exceed the cardinality cap are dropped and counted rather than failing the batch, since one runaway label should not blind the operator to everything else.

func (*Store) Stats 

func (s *Store) Stats() Stats

Stats returns a snapshot of the store's bookkeeping.

func (*Store) Summary 

func (s *Store) Summary() []SeriesSummary

Summary reports every cumulative series, sorted by metric then tags so the output is stable between calls and diffable between runs.

func (*Store) Timeline 

func (s *Store) Timeline(since time.Time) []Bucket

Timeline returns finalised buckets starting at or after `since`, oldest first. A zero `since` returns the whole retained window.

func (*Store) Totals 

func (s *Store) Totals() map[string]SeriesSummary

Totals returns one merged aggregate per metric, folded across every label combination.

This is the only correct way to get a whole-run figure. Series are stored per label set — per endpoint, per status code — and the summary statistics of those slices cannot simply be averaged back together: a mean must be re-weighted by count, and a percentile has to be recomputed from the merged distribution. Callers that fold Summary() themselves get plausible-looking numbers that disagree with the thresholds, which is worse than no numbers.

type StoreConfig 

type StoreConfig struct {
	// Resolution is the width of one time bucket. Zero applies one second.
	Resolution time.Duration

	// Window is how much history is retained for charting. Older buckets are
	// evicted. Zero applies one hour.
	Window time.Duration

	// LateGrace is how long a bucket accepts further contributions before it
	// is finalised. It has to cover the worst-case skew between an agent
	// flushing and the coordinator receiving; too short and slow nodes get
	// silently dropped from the tail of the chart.
	LateGrace time.Duration

	// Histogram must match the resolution every node is recording at.
	Histogram HistogramConfig

	// MaxSeries caps distinct cumulative series held. Nodes enforce their own
	// cap, but a large fleet can still sum to more than the coordinator
	// should hold, so it is enforced again here.
	MaxSeries int

	// MaxChartedEndpoints caps how many request names get their own line in
	// the time buckets. Zero applies DefaultMaxChartedEndpoints.
	MaxChartedEndpoints int

	// MaxFailureKinds caps distinct kinds of failure retained for the run.
	MaxFailureKinds int
}

StoreConfig tunes the coordinator's view of a run.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.