otelbeat

README

otelbeat

OpenTelemetry metrics for scheduled jobs running with beat. It implements beat.Handler, which beat calls once per run.

• Two instruments, two attributes - enough to answer rate, errors, latency and throughput, few enough that cardinality stays constant.
• The application owns the pipeline - otelbeat records, it does not configure, aggregate or export.

go get github.com/uchaloop/otelbeat

Quick start

fx.New(
	// Provide your metric.MeterProvider and exporter.
	otelbeatfx.Module(),
	beatfx.Module(),
)

Without Fx:

handler, err := otelbeat.New(meterProvider.Meter("beat"))

What it records

Instrument	Type	What it answers
beat.run.duration	float64 histogram, seconds	Duration, and run count through its own count
beat.processed	int64 counter, items	Throughput

Both carry status (ok, error, panic) and mode (interval, cron).

rate(beat_run_duration_seconds_count[5m])                    # runs
rate(beat_run_duration_seconds_count{status="error"}[5m])    # failures
histogram_quantile(0.95, sum by (le) (rate(beat_run_duration_seconds_bucket[5m])))
rate(beat_processed_total[5m])                               # items

Documentation

What each instrument means, how a run is classified, and why the attributes stop at two are in the package documentation: pkg.go.dev/github.com/uchaloop/otelbeat.

Acknowledgements

I am grateful to the authors of OpenTelemetry Go. Their work made this library possible.

License

MIT

Documentation

Overview

Package otelbeat records OpenTelemetry metrics for a beat.Beat. It implements beat.Handler, which beat calls once per run with a Record, and turns that into two instruments on a meter the application supplies.

handler, err := otelbeat.New(meterProvider.Meter("beat"))

The otelbeatfx subpackage wires the same handler into an Uber Fx application.

What is recorded

• beat.run.duration - a float64 histogram in seconds. Its count is also the number of runs, so duration and rate come from one instrument rather than a histogram plus a counter that could disagree.
• beat.processed - an int64 counter of the items a run reported processing.

Both carry two attributes, and only two:

• status - "ok", "error" or "panic";
• mode - "interval" or "cron", the scheduling beat is running under.

That is enough for one query sliced by those labels to answer success and error counts, latency and throughput per scheduling mode, and few enough that the cardinality stays a constant rather than a function of the workload.

The "panic" status appears only when beat's recovery middleware is installed: it is what converts a panic into a *beat.PanicError for the Record to carry. Without it a panic crashes the process, which is beat's deliberate default, and there is no run to record.

Classifying a run

DefaultStatus maps a Record to one of the three values above. WithStatus replaces it, for an application whose errors deserve to be told apart:

handler, err := otelbeat.New(
	meter,
	otelbeat.WithStatus(func(record beat.Record) string {
		switch {
		case errors.Is(record.Err, ErrThrottled):
			return "throttled"
		default:
			return otelbeat.DefaultStatus(record)
		}
	}),
)

Keep the set of values small and closed. Every distinct status is a time series per instrument, so one derived from an error message rather than from a sentinel turns a metric into a cardinality incident.

What it does not do

The application owns the MeterProvider and whatever exports from it. otelbeat records; it does not configure, aggregate or export, and it holds no opinion about where the metrics go.

Index

• func DefaultStatus(r beat.Record) string
• func New(meter metric.Meter, opts ...Option) (beat.Handler, error)
• type Option
  • func WithStatus(f StatusFunc) Option
• type StatusFunc

Functions

func DefaultStatus

func DefaultStatus(r beat.Record) string

DefaultStatus classifies a run as "ok", "panic" (a recovered panic - a *beat.PanicError) or "error". Build on it in a custom StatusFunc.

The "panic" status only appears when beat's recovery middleware is in use; without it a panicking Job crashes the process and produces no Record.

func New

func New(meter metric.Meter, opts ...Option) (beat.Handler, error)

New builds a beat.Handler that records the instruments described in the package doc on meter. The application owns the meter and its exporter.

Types

type Option

type Option func(*settings)

Option configures the handler built by New.

func WithStatus

func WithStatus(f StatusFunc) Option

WithStatus overrides how a run is classified into the "status" attribute. The default is DefaultStatus. A nil func is ignored. Keep the set of returned values small - each distinct value is a separate metric series.

type StatusFunc

type StatusFunc func(beat.Record) string

StatusFunc classifies a run into the value of the "status" attribute. Use it to map domain errors to your own codes; DefaultStatus gives "ok", "error" or "panic".