otelbeat

README

otelbeat

OpenTelemetry metrics for scheduled jobs running with beat.

Installation

go get github.com/uchaloop/otelbeat

Fx

Provide a metric.MeterProvider, then install the metrics handler before the beat module:

fx.New(
	// Provide your metric.MeterProvider and exporter.
	otelbeatfx.Module(),
	beatfx.Module(),
)

Without Fx

handler, err := otelbeat.New(
	meterProvider.Meter("beat"),
)

The returned value implements beat.Handler.

Metrics

Instrument	Type	Description
beat.run.duration	Float64 histogram, seconds	Job duration and run count
beat.processed	Int64 counter, items	Number of processed items

Both instruments include:

• status: ok, error, or panic;
• mode: interval or cron.

The panic status is available when beat's recovery middleware converts the panic into *beat.PanicError.

Custom status

Map application errors to custom status values:

handler, err := otelbeat.New(
	meter,
	otelbeat.WithStatus(func(record beat.Record) string {
		switch {
		case errors.Is(record.Err, ErrThrottled):
			return "throttled"
		default:
			return otelbeat.DefaultStatus(record)
		}
	}),
)

Keep the number of status values small to avoid high-cardinality metrics.

Prometheus queries

Run rate:

rate(beat_run_duration_seconds_count[5m])

Error rate:

rate(beat_run_duration_seconds_count{status="error"}[5m])

P95 duration:

histogram_quantile(0.95, sum by (le) (rate(beat_run_duration_seconds_bucket[5m])))

Processing throughput:

rate(beat_processed_total[5m])

Acknowledgements

I am grateful to the authors of OpenTelemetry Go. Their work made this library possible.

License

MIT

Documentation

Overview

Package otelbeat records OpenTelemetry metrics for a beat.Beat. It implements beat.Handler and, per run, records two instruments on a meter the application supplies:

• beat.run.duration (histogram, seconds) - its count also gives the number of runs;
• beat.processed (counter) - items processed, summed.

Both carry the attributes "status" and "mode", so a single query sliced by those labels covers success/error/panic counts, latency and throughput per scheduling mode. The application owns the MeterProvider and its exporter/reader; otelbeat records, it does not export.

Index

• func DefaultStatus(r beat.Record) string
• func New(meter metric.Meter, opts ...Option) (beat.Handler, error)
• type Option
  • func WithStatus(f StatusFunc) Option
• type StatusFunc

Functions

func DefaultStatus

func DefaultStatus(r beat.Record) string

DefaultStatus classifies a run as "ok", "panic" (a recovered panic - a *beat.PanicError) or "error". Build on it in a custom StatusFunc.

The "panic" status only appears when beat's recovery middleware is in use; without it a panicking Job crashes the process and produces no Record.

func New

func New(meter metric.Meter, opts ...Option) (beat.Handler, error)

New builds a beat.Handler that records the instruments described in the package doc on meter. The application owns the meter and its exporter.

Types

type Option

type Option func(*settings)

Option configures the handler built by New.

func WithStatus

func WithStatus(f StatusFunc) Option

WithStatus overrides how a run is classified into the "status" attribute. The default is DefaultStatus. A nil func is ignored. Keep the set of returned values small - each distinct value is a separate metric series.

type StatusFunc

type StatusFunc func(beat.Record) string

StatusFunc classifies a run into the value of the "status" attribute. Use it to map domain errors to your own codes; DefaultStatus gives "ok", "error" or "panic".