metrics

Documentation

Overview

Package metrics exposes OpticTrace telemetry to Prometheus.

Design notes:

• The collector owns a private Registry (not the global default), so embedding OpticTrace in an app that already uses client_golang can never cause duplicate-registration panics.
• Custom labels from optic.yaml become real Prometheus dimensions, taken from engine.LabelKeys(); requests without a value export "" (Prometheus requires stable schemas). A Prometheus metric cannot change its label set, so a config reload that adds a label REBUILDS the affected vectors (see SetLabelKeys) rather than mutating them. Without that, a newly added label would show up in the dashboard but silently never appear in /metrics — a Grafana panel that stays empty with no error anywhere.
• The `route` label carries the matched rule's glob (or a normalized path), never the raw path — cardinality stays bounded by design.
• P50/P95/P99 are derived by Prometheus from the latency histogram, e.g.: histogram_quantile(0.99, sum by (le, route) (rate(optictrace_request_duration_seconds_bucket[5m])))

Index

• Constants
• type Collector
  • func New(service string, buckets []float64, customKeys []string, maxLabelValues int) *Collector
  • func (c *Collector) AppLogDropped(reason string, n int)
  • func (c *Collector) AppLogStored(n int)
  • func (c *Collector) ExportDelivered(exporter string, n int)
  • func (c *Collector) ExportDropped(exporter string)
  • func (c *Collector) ExportFailed(exporter string, n int)
  • func (c *Collector) Handler() http.Handler
  • func (c *Collector) InflightDec()
  • func (c *Collector) InflightInc()
  • func (c *Collector) LabelKeys() []string
  • func (c *Collector) Observe(o Observation)
  • func (c *Collector) SDKIngested()
  • func (c *Collector) SetLabelKeys(keys []string) bool
  • func (c *Collector) SpanObserved(name, kind, service string, seconds float64)
  • func (c *Collector) SpansDropped(reason string, n int)
  • func (c *Collector) SpansStored(n int)
  • func (c *Collector) StoreDropped()
  • func (c *Collector) StreamClosed()
  • func (c *Collector) StreamOpened()
• type Observation

Constants

const OverLimit = "__over_limit__"

OverLimit replaces a custom label's value once that label has already contributed maxLabelValues distinct values. Every subsequent unseen value collapses into this single series, so cardinality stops growing.

Types

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector aggregates per-request metrics and serves them via Handler().

func New

func New(service string, buckets []float64, customKeys []string, maxLabelValues int) *Collector

New builds a Collector for one service. customKeys is the fixed set of optic.yaml label names (from engine.LabelKeys()); maxLabelValues caps the distinct values each may contribute (0 disables the guard).

func (*Collector) AppLogDropped

func (c *Collector) AppLogDropped(reason string, n int)

AppLogDropped counts a discarded line under the reason it was discarded.

func (*Collector) AppLogStored

func (c *Collector) AppLogStored(n int)

AppLogStored counts lines persisted against a span.

func (*Collector) ExportDelivered

func (c *Collector) ExportDelivered(exporter string, n int)

Exporter accounting — satisfies export.Metrics.

func (*Collector) ExportDropped

func (c *Collector) ExportDropped(exporter string)

func (*Collector) ExportFailed

func (c *Collector) ExportFailed(exporter string, n int)

func (*Collector) Handler

func (c *Collector) Handler() http.Handler

Handler serves the /metrics endpoint in Prometheus exposition format.

func (*Collector) InflightDec

func (c *Collector) InflightDec()

func (*Collector) InflightInc

func (c *Collector) InflightInc()

InflightInc/Dec bracket a proxied request.

func (*Collector) LabelKeys

func (c *Collector) LabelKeys() []string

LabelKeys returns the custom label schema currently in effect.

func (*Collector) Observe

func (c *Collector) Observe(o Observation)

Observe records one completed exchange. Safe for concurrent use.

func (*Collector) SDKIngested

func (c *Collector) SDKIngested()

SDKIngested counts a record received from a framework SDK.

func (*Collector) SetLabelKeys

func (c *Collector) SetLabelKeys(keys []string) bool

SetLabelKeys re-points the config-dependent metrics at a new label schema, returning true if anything changed.

A Prometheus metric's label set is immutable — and a Registry refuses to re-register a name under a different schema even after Unregister — so this swaps in a freshly built registry holding freshly built vectors. Request counts and latency buckets therefore restart from zero, and the old series go stale.

That is the correct trade. Before this existed, adding a `labels:` key and reloading made the label appear in the dashboard while never appearing in /metrics: a Grafana panel built on it returned no data, with no error anywhere to explain why. Only the two config-dependent vectors are affected; every other counter lives in the stable registry and survives.

func (*Collector) SpanObserved

func (c *Collector) SpanObserved(name, kind, service string, seconds float64)

SpanObserved records one operation's duration, so "which query is slow" can be alerted on rather than only browsed.

func (*Collector) SpansDropped

func (c *Collector) SpansDropped(reason string, n int)

SpansDropped counts discarded spans under the reason they were discarded. request_cap being the reason is itself a finding — it means one request produced more operations than the policy allows, which is usually an N+1.

func (*Collector) SpansStored

func (c *Collector) SpansStored(n int)

SpansStored counts inner spans persisted against a request.

func (*Collector) StoreDropped

func (c *Collector) StoreDropped()

StoreDropped counts a telemetry record lost to backpressure.

func (*Collector) StreamClosed

func (c *Collector) StreamClosed()

func (*Collector) StreamOpened

func (c *Collector) StreamOpened()

StreamOpened/Closed bracket a streaming response, so a live SSE connection is visible while it is open rather than only once it ends.

type Observation

type Observation struct {
	// Service names the application the exchange belongs to. Empty means the
	// agent's own service — the sidecar case, where they are the same thing.
	// An SDK reporting into a shared agent sets it, which is what keeps a
	// fleet from collapsing into one series.
	Service   string
	Method    string
	Route     string // low-cardinality route pattern, not the raw path
	Status    int
	Duration  time.Duration
	ReqBytes  int64
	RespBytes int64
	Labels    map[string]string // values for the custom label schema
	// Stream marks a long-lived response (SSE, chunked streaming) rather
	// than a request/response exchange. Its duration is recorded on a
	// separate histogram: a 10-minute stream observed as a 600s request
	// makes the route's p95 meaningless for the whole window.
	Stream bool
}

Observation is one completed HTTP exchange, as seen by the collector.