otelbeat

README

otelbeat

OpenTelemetry metrics for beat. The package implements beat.Handler; the application supplies a Meter and owns the SDK, Resource, exporter and shutdown. Requires Go 1.27 or later.

Usage

Given an application-configured metric.MeterProvider and a beat.Job:

handler, err := otelbeat.New(provider.Meter("beat"))
if err != nil {
    return err
}
runner, err := beat.MakeBeat(beat.Config{
    Period: 5 * time.Minute, JobTimeout: 4 * time.Minute,
    Jitter: 5 * time.Minute,
}, job, handler)
if err != nil {
    return err
}

Start and stop the runner through its lifecycle. Stop beat before shutting down the MeterProvider so the final completion can be recorded and exported. To add logging, supply beat.MultiHandler(handler, logHandler).

flowchart LR
    J[Job returns] --> R[beat.Record]
    R --> H[otelbeat Handler]
    H --> M[Meter: record measurements]
    M --> S[Application SDK and Views]
    S --> E[Reader and exporter]
    E --> B[Metrics backend]

Compilable examples of provider wiring and custom outcomes are in example_test.go.

Instruments

Name	Type / unit	Measurement	Attributes
beat.run.duration	Histogram / seconds	Job elapsed time, including middleware	mode, outcome
beat.run.saturation	Histogram / ratio	Duration / Period	mode, outcome
beat.run.lateness	Histogram / seconds	max(0, Start − ScheduledFor)	mode, outcome
beat.processed	Counter / items	Nonnegative Job-reported processed count	mode, outcome
beat.missed	Counter / points	Previously computed unintentional grid losses	mode

The duration histogram count counts completed runs. No completion is recorded while a Job is still running. Duration excludes Handler and backoff time.

Default outcomes are ok, error, panic, timeout, and canceled, taken from Record.Outcome; an empty outcome becomes unknown. Modes are fixed_rate and fixed_delay. Panic records require beat's recovery middleware.

beat.missed excludes points bypassed by intentional backoff. Its count is reported with the next completed Job, so shutdown or a stuck Job can prevent its delivery. It has no outcome attribute because the reporting run's outcome does not describe the lost points. In fixed-delay mode Missed is always zero, and saturation compares work duration with the configured rest interval.

A slow Handler can cause grid losses without increasing the next run's lateness. Neither saturation nor Missed alone determines whether to add replicas: also observe queue depth, oldest-item age, arrival rate and dependency capacity. An unchanged completion count can suggest a stuck Job only when the expected maximum interval without completions is known; backoff and telemetry failure can produce the same signal. Cumulative histogram samples may still be exported.

Histogram boundaries

Boundaries are advisory; an SDK View can override them.

Histogram	Boundaries
Duration (s)	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 300, 600
Saturation	0.1, 0.25, 0.5, 0.7, 0.8, 0.9, 1, 1.1, 1.25, 1.5, 2, 4
Lateness (s)	0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60

Values above the last boundary still contribute to the overflow bucket. For manually constructed Records, a nonpositive Period omits saturation and a zero ScheduledFor omits lateness. Negative processed and missed counts are clamped to zero.

Custom outcomes

Keep labels to a small, fixed set. Preserve timeout, cancellation and panic classifications when adding domain-specific error categories:

handler, err := otelbeat.New(meter,
    otelbeat.WithOutcome(func(r beat.Record) string {
        if r.Outcome == beat.OutcomeError && errors.Is(r.Err, errThrottled) {
            return "throttled"
        }
        return otelbeat.DefaultOutcome(r)
    }),
)

Here errThrottled is an application sentinel. Do not use error messages or item identifiers as outcome labels.

Replicas and Fx

Set service and instance identity on the application's OTel Resource, for example service.name, service.instance.id and k8s.cluster.name. Ensure the exporter or collector preserves the identity needed by your backend; Resource attributes are not necessarily copied into every metric's labels. otelbeat does not modify the Resource. Aggregate processed rates across the intended service's replicas.

Given provider with static type metric.MeterProvider, cfg and job:

fx.New(
    fx.Provide(func() metric.MeterProvider { return provider }),
    fx.Supply(cfg),
    fx.Provide(func() beat.Job { return job }),
    otelbeatfx.Module(),
    beatfx.Module(),
)

The application must start/stop the Fx app and arrange provider shutdown after beat. otelbeatfx.Module supplies a Handler on the "beat" meter.

Example queries

For a Prometheus pipeline using the names below, filter to the intended service and cluster. Exporter naming and resource-label mappings can differ.

# Completed runs and processed items per second.
sum(rate(beat_run_duration_seconds_count[30m]))
sum(rate(beat_processed_total[30m]))

# Completed runs classified as failures.
sum(rate(beat_run_duration_seconds_count{outcome=~"error|panic|timeout"}[30m]))

# p95 ratio of Job duration to Period, grouped by scheduling mode.
histogram_quantile(0.95,
  sum by (le, mode) (rate(beat_run_saturation_bucket[30m])))

# Unintentional grid losses per second.
sum(rate(beat_missed_total[30m]))

Choose a query window with enough completions for your configured period.

API reference · MIT license

Documentation

Overview

Package otelbeat implements beat.Handler using OpenTelemetry metrics. New accepts an application-provided Meter. The application owns the SDK, Resource, Views, reader/exporter and shutdown; stop beat before the provider.

Instruments

beat.run.duration records Job duration in seconds; its count counts completed runs. beat.run.saturation records Duration / Period. beat.run.lateness records nonnegative Start minus ScheduledFor in seconds. beat.processed accumulates Job-reported items. These instruments carry mode and outcome attributes. DefaultOutcome uses Record.Outcome, or "unknown" when it is empty. WithOutcome supplies a custom classifier; keep its output to a small fixed set.

beat.missed accumulates unintentional grid losses and carries mode alone. Intentional backoff is excluded. Losses arrive with the next completed Job, so shutdown or a stuck Job can prevent their delivery. Missed is zero in fixed-delay mode. The current run's outcome cannot identify the loss's cause.

All histograms have explicit advisory boundaries, overridable by SDK Views. Nonpositive Period omits saturation; zero ScheduledFor omits lateness. Negative processed and missed counts are clamped to zero.

Interpretation

Metrics describe execution, not queue capacity. Scaling decisions also need queue depth, oldest-item age, arrival rate and dependency capacity. A slow Handler can cause missed points without lateness. An unchanged completion count suggests a stuck Job only with a known maximum interval between completions; backoff and telemetry failure can produce the same signal.

Put service, instance and cluster identity on the application's Resource and configure the export pipeline to retain the distinctions needed in queries. otelbeat does not modify Resource attributes. The otelbeatfx package provides an Fx integration. See the README for examples, boundaries and queries.

Index

• func DefaultOutcome(r beat.Record) string
• func New(meter metric.Meter, opts ...Option) (beat.Handler, error)
• type Option
  • func WithOutcome(f OutcomeFunc) Option
• type OutcomeFunc

Examples

• New
• WithOutcome

Functions

func DefaultOutcome

func DefaultOutcome(r beat.Record) string

DefaultOutcome reports the classification beat made in Record.Outcome: "ok", "error", "panic", "timeout" or "canceled". Build on it in a custom OutcomeFunc.

The "panic" outcome only appears when beat's recovery middleware is in use; without it a panicking Job crashes the process and produces no Record. A Record that carries no outcome at all - one assembled by hand rather than by beat - reads as "unknown" rather than as an empty label.

func New

func New(meter metric.Meter, opts ...Option) (beat.Handler, error)

New builds a beat.Handler that records the instruments described in the package doc on meter. The application owns the meter and its exporter.

Example

package main

import (
	"context"
	"time"

	"github.com/uchaloop/beat"
	"github.com/uchaloop/otelbeat"

	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

func main() {
	// ManualReader permits explicit collection. A production application can
	// instead supply its configured reader/exporter to the MeterProvider.
	reader := sdkmetric.NewManualReader()
	provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))
	handler, err := otelbeat.New(provider.Meter("beat"))
	if err != nil {
		panic(err)
	}
	runner, err := beat.MakeBeat(beat.Config{
		Period: 5 * time.Minute, JobTimeout: 4 * time.Minute,
		Jitter: 5 * time.Minute,
	}, func(ctx context.Context) (int, error) { return 0, ctx.Err() }, handler)
	if err != nil {
		panic(err)
	}
	if err := runner.Start(context.Background()); err != nil {
		panic(err)
	}
	// The application runs until its shutdown signal.
	stopCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	if err := runner.Stop(stopCtx); err != nil {
		panic(err)
	}
	exportCtx, exportCancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer exportCancel()
	if err := provider.Shutdown(exportCtx); err != nil {
		panic(err)
	}
}

Types

type Option

type Option func(*settings)

Option configures the handler built by New.

func WithOutcome

func WithOutcome(f OutcomeFunc) Option

WithOutcome overrides how a run is classified into the "outcome" attribute. The default is DefaultOutcome. A nil func is ignored. Keep the set of returned values small - each distinct value is a separate metric series.

Example

package main

import (
	"context"
	"errors"

	"github.com/uchaloop/beat"
	"github.com/uchaloop/otelbeat"

	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

func main() {
	errThrottled := errors.New("throttled")
	reader := sdkmetric.NewManualReader()
	provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))
	defer provider.Shutdown(context.Background())
	handler, err := otelbeat.New(provider.Meter("beat"),
		otelbeat.WithOutcome(func(r beat.Record) string {
			if r.Outcome == beat.OutcomeError && errors.Is(r.Err, errThrottled) {
				return "throttled"
			}
			return otelbeat.DefaultOutcome(r)
		}),
	)
	if err != nil {
		panic(err)
	}
	_ = handler // Supply this handler to beat.MakeBeat.
}

type OutcomeFunc

type OutcomeFunc func(beat.Record) string

OutcomeFunc classifies a run into the value of the "outcome" attribute. Use it to map domain errors to your own codes; DefaultOutcome passes through what beat already decided.