otelbeat

README

otelbeat

OpenTelemetry metrics for beat, the background-job scheduler. otelbeat implements beat.Handler and records a minimal, attribute-driven set of instruments per run on a meter the application supplies.

The beat core has no metrics - it hands each run to a Handler. otelbeat is that handler when you want OpenTelemetry. The dependency is one-directional (otelbeat -> beat); the core never imports OpenTelemetry.

Install

go get github.com/uchaloop/otelbeat

Instruments

Two instruments, recorded per run:

Instrument	Type	Typical Prometheus series
beat.run.duration	Float64Histogram (s)	beat_run_duration_seconds_bucket / _sum / _count
beat.processed	Int64Counter ({item})	beat_processed_total

The histogram's _count also gives the number of runs, so run/error/panic counts and latency come from one instrument.

Attributes

Both instruments carry:

• status - ok / error / panic (or your own codes, see WithStatus)
• mode - interval / cron

status(3) × mode(2) = 6 series per instrument - slice in Grafana, no metric explosion.

The panic status requires beat's recovery middleware. Without it a panicking Job crashes the process and records nothing.

Usage

The application owns the MeterProvider and its exporter (Prometheus, OTLP, …); otelbeat only records.

With Fx, otelbeatfx.Module provides the beat.Handler from the container's metric.MeterProvider:

import "github.com/uchaloop/otelbeat/otelbeatfx"

fx.New(
    // ... your MeterProvider ...
    otelbeatfx.Module(),
    beatfx.Module(),
)

Without Fx (or to customize), call otelbeat.New directly:

handler, err := otelbeat.New(meterProvider.Meter("beat"))

The core package github.com/uchaloop/otelbeat has no Fx dependency; the Fx integration lives in github.com/uchaloop/otelbeat/otelbeatfx.

Map domain errors to your own status codes with WithStatus, building on otelbeat.DefaultStatus:

otelbeat.New(meter, otelbeat.WithStatus(func(r beat.Record) string {
    var pe *beat.PanicError
    switch {
    case errors.As(r.Err, &pe):
        return "panic"
    case errors.Is(r.Err, ErrThrottled):
        return "2"
    case r.Err != nil:
        return "1"
    default:
        return "0"
    }
}))

Example queries

rate(beat_run_duration_seconds_count[5m])                                   # run rate
rate(beat_run_duration_seconds_count{status="error"}[5m])                   # error rate
histogram_quantile(0.95, sum by(le)(rate(beat_run_duration_seconds_bucket[5m])))  # p95 latency
rate(beat_processed_total[5m])                                              # throughput

Acknowledgements

otelbeat builds on OpenTelemetry Go. Thanks to its authors and maintainers.

License

MIT.

Documentation

Overview

Package otelbeat records OpenTelemetry metrics for a beat.Beat. It implements beat.Handler and, per run, records two instruments on a meter the application supplies:

• beat.run.duration (histogram, seconds) - its count also gives the number of runs;
• beat.processed (counter) - items processed, summed.

Both carry the attributes "status" and "mode", so a single query sliced by those labels covers success/error/panic counts, latency and throughput per scheduling mode. The application owns the MeterProvider and its exporter/reader; otelbeat records, it does not export.

Index

• func DefaultStatus(r beat.Record) string
• func New(meter metric.Meter, opts ...Option) (beat.Handler, error)
• type Option
  • func WithStatus(f StatusFunc) Option
• type StatusFunc

Functions

func DefaultStatus

func DefaultStatus(r beat.Record) string

DefaultStatus classifies a run as "ok", "panic" (a recovered panic - a *beat.PanicError) or "error". Build on it in a custom StatusFunc.

The "panic" status only appears when beat's recovery middleware is in use; without it a panicking Job crashes the process and produces no Record.

func New

func New(meter metric.Meter, opts ...Option) (beat.Handler, error)

New builds a beat.Handler that records the instruments described in the package doc on meter. The application owns the meter and its exporter.

Types

type Option

type Option func(*settings)

Option configures the handler built by New.

func WithStatus

func WithStatus(f StatusFunc) Option

WithStatus overrides how a run is classified into the "status" attribute. The default is DefaultStatus. A nil func is ignored. Keep the set of returned values small - each distinct value is a separate metric series.

type StatusFunc

type StatusFunc func(beat.Record) string

StatusFunc classifies a run into the value of the "status" attribute. Use it to map domain errors to your own codes; DefaultStatus gives "ok", "error" or "panic".