oida

README

oida - in process telemetry for Go services

Oida is a Go importable package that implements in-process telemetry and provides a rich interface for observation. Its goal is to be an observability platform for small and big projects that benefit from local observability. The project started as an extension of titpetric/phpscript which got separated out to handle more concerns than just server status. The name of the project comes from my Austrian-roots friend Martin. The word itself is likely to mean various things depending on context; whole conversations can be had just by repeating the words in different tone.

Docs | Install | Import | Use with stdlib | Use with go-chi | Integration | Features

Install

go get github.com/titpetric/oida@latest

Import

import "github.com/titpetric/oida"

Use with stdlib

testdata/examples/std/main_std.go is a service on *http.ServeMux: oida.New builds the tracer, oida.Mount registers the dashboard, and tracer.Middleware wraps the mux, so every sampled request through it is recorded. Options.Path is added to IgnorePaths, so browsing the dashboard records nothing of its own.

Use with go-chi

testdata/examples/chi/main_chi.go is the same service on a chi.Router, which takes the middleware through r.Use(tracer.Middleware). Register it before the routes it records: chi panics on a Use that follows a route.

oida.Mount serves the dashboard under the path the tracer was configured with, and takes a chi.Router or an *http.ServeMux alike. A router whose Handle returns a value, such as gorilla's, mounts through oida.RouterFunc; testdata/examples/gorilla/main_gorilla_mux.go is that wiring, and the getting started guide explains it. The tracer is an http.Handler of its own, so mux.Handle("/debug/oida/", tracer) works where one pattern is enough.

Recording is opt-in: set opts.Enabled in code, or leave it alone and set OIDA_ENABLED=true in the environment. Open http://localhost:8080/debug/oida.

Integration

getUser is the handler each example registers, and listUsers under it records the span:

_, span := oida.Start(ctx, "SELECT users", oida.KindDatabase)
defer span.End()

span.SetAttribute("limit", 100)

Pass the context down, and the next oida.Start records a child span under this one. The kind drives the colour in the timeline and the grouping of the segment sweep; the span kinds and the attribute keys the dashboard reads are documented with the rest of the data model. Every call is nil-safe, so instrumented code runs unchanged where no tracer was built, or where the request was not sampled.

Features

• HTTP requests and background jobs
• Nested spans with kinds, attributes, errors, and source locations
• Bounded in-memory retention, or disk documents that outlive the process, chosen in code or from the environment
• Rate sampling and custom sampling rules
• Live activity, retained traces, route statistics, and trace details
• HTML for browsers, JSON for tools, and plain text for terminals
• Nil-safe instrumentation when tracing is absent or a request is not sampled

github.com/titpetric/oida is the public API, and docs/api.md is its generated reference. The frontend, model and storage packages serve the root package and carry no compatibility promise of their own. docs/ covers getting started, instrumentation, configuration, the data model and the dashboard, and testdata/examples/ holds a runnable program per router.

License

MIT, copyright Tit Petric.

Documentation

Overview

Package oida records in-process telemetry: traces and spans held in a ring buffer inside the process, with a server side rendered front end mounted at /debug/oida.

Wire it into a service in three calls: configure the tracer, mount it, add the middleware. Recording is opt-in: enable it in code, or leave the field alone and set OIDA_ENABLED=true in the environment. The tracer is an http.Handler serving the debug front end, so it mounts like any other handler and no second import is needed:

opts := oida.NewOptions("billing-api")
opts.Enabled = true

tracer, err := oida.New(opts)
if err != nil {
	return err
}

mux := http.NewServeMux()
mux.Handle("/debug/oida/", tracer)

A chi router mounts the same tracer with its own call:

r := chi.NewRouter()
r.Mount("/debug/oida", tracer)

Mount registers the front end on either router, adding the subtree patterns each one understands:

if err := oida.Mount(mux, tracer); err != nil {
	return err
}

The middleware records every sampled request into the tracer:

handler := tracer.Middleware(mux)
return http.ListenAndServe(":8080", handler)

Instrument anything below the middleware:

ctx, span := oida.Start(ctx, "SELECT users", oida.KindDatabase)
defer span.End()
span.SetAttribute("limit", limit)

Every instrumentation call is nil safe, so instrumented code runs unchanged in processes where oida is disabled, where the request was not sampled, or where no trace is in the context.

The project is four packages. This one records and serves: the tracer, the middleware and the options. Package model holds the recorded data and the configuration and depends on nothing; the types it defines are aliased here, so instrumenting a service needs this import alone. Package storage holds the retention drivers, which New builds from the environment. Package frontend renders the dashboard, reads the model alone, and is imported here so the tracer can serve it.

Nothing in this package writes to stdout or stderr. Storage and rendering failures are reported through Options.OnError.

Index

• Constants
• Variables
• func Do(ctx context.Context, name string, fn func(context.Context) error, kind ...Kind) error
• func Mount(r Router, t *Tracer) error
• func RecordError(ctx context.Context, err error)
• func TraceID(ctx context.Context) string
• func WithTrace(ctx context.Context, t *Trace) context.Context
• type Attributes
• type Auth
  • func NewAuth(opts Options) (*Auth, error)
• type Kind
• type LogEntry
• type Options
  • func NewOptions(serviceName string) Options
• type Router
• type RouterFunc
  • func (f RouterFunc) Handle(pattern string, h http.Handler)
• type Sampler
• type Snapshot
• type Span
  • func SpanFromContext(ctx context.Context) *Span
  • func Start(ctx context.Context, name string, kind ...Kind) (context.Context, *Span)
  • func StartAuto(ctx context.Context, symbol any, kind ...Kind) (context.Context, *Span)
  • func StartRequest(r *http.Request, name string, kind ...Kind) (*http.Request, *Span)
  • func StartSpan(ctx context.Context, name string, kind ...Kind) *Span
• type State
• type Storage
• type Trace
  • func TraceFromContext(ctx context.Context) *Trace
• type Tracer
  • func New(opts Options) (*Tracer, error)
  • func (t *Tracer) Enabled() bool
  • func (t *Tracer) Finish(trace *Trace)
  • func (t *Tracer) Live() []Trace
  • func (t *Tracer) Middleware(next http.Handler) http.Handler
  • func (t *Tracer) Observe(ctx context.Context, name string, fn func(context.Context) error) error
  • func (t *Tracer) Options() Options
  • func (t *Tracer) ReportError(err error)
  • func (t *Tracer) Reset()
  • func (t *Tracer) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (t *Tracer) SetEnabled(enabled bool)
  • func (t *Tracer) Snapshot() Snapshot
  • func (t *Tracer) StartTrace(ctx context.Context, name string) (context.Context, *Trace, error)
  • func (t *Tracer) Subscribe() (<-chan struct{}, func())
  • func (t *Tracer) Trace(id string) (Trace, bool)
  • func (t *Tracer) Traces() []Trace

Constants

const (
	KindInternal = model.KindInternal
	KindHTTP     = model.KindHTTP
	KindDatabase = model.KindDatabase
	KindExternal = model.KindExternal
	KindTemplate = model.KindTemplate
	KindCache    = model.KindCache
	KindQueue    = model.KindQueue
)

Span kinds. The set is open: an unrecognized value is valid.

const (
	AttrMemoryLimit = model.AttrMemoryLimit
	AttrMemoryUsage = model.AttrMemoryUsage
)

Well known attribute keys. The set is open; these are the ones the front end renders as sizes.

const (
	StateWaiting    = model.StateWaiting
	StateStarting   = model.StateStarting
	StateReading    = model.StateReading
	StateProcessing = model.StateProcessing
	StateWriting    = model.StateWriting
	StateKeepalive  = model.StateKeepalive
	StateClosing    = model.StateClosing
	StateError      = model.StateError
)

Scoreboard states of a trace in flight.

const (
	LevelInfo  = model.LevelInfo
	LevelError = model.LevelError
)

Log levels recorded by Trace.Info and Trace.Error.

const DefaultPath = model.DefaultPath

DefaultPath is the default mount path of the debug front end.

const RequestIDHeader = model.RequestIDHeader

RequestIDHeader carries the trace identifier on the request and the response.

const SessionCookie = model.SessionCookie

SessionCookie is the name of the front end session cookie.

const SessionTTL = model.SessionTTL

SessionTTL is how long an issued session token stays valid.

Variables

var (
	// ErrInvalidCredentials is returned when a login does not match any
	// configured user.
	ErrInvalidCredentials = model.ErrInvalidCredentials

	// ErrInvalidToken is returned when a session cookie or bearer token does
	// not verify against the signing secret, or has expired.
	ErrInvalidToken = model.ErrInvalidToken
)

The errors authentication returns, aliased like the other error values so errors.Is works with either spelling.

var (
	// ErrNilRouter is returned when Mount is called without a router.
	ErrNilRouter = model.ErrNilRouter

	// ErrNoTracer is returned when Mount is called without a tracer, which is
	// a dashboard with nothing to show.
	ErrNoTracer = model.ErrNoTracer

	// ErrInvalidOptions is the base error for every configuration failure.
	ErrInvalidOptions = model.ErrInvalidOptions

	// ErrInvalidPath is returned when Options.Path is not an absolute path.
	ErrInvalidPath = model.ErrInvalidPath

	// ErrInvalidSampleRate is returned when Options.SampleRate is outside
	// [0,100].
	ErrInvalidSampleRate = model.ErrInvalidSampleRate

	// ErrTraceNotFound is returned when a trace ID is not in the ring buffer.
	ErrTraceNotFound = model.ErrTraceNotFound

	// ErrDisabled is returned when a trace is requested from a disabled tracer.
	ErrDisabled = model.ErrDisabled
)

The errors this package returns. Every configuration failure wraps ErrInvalidOptions, so a caller can test for the class or for the case. The values live in the model package so the front end can return them too; these are the same error values, so errors.Is works with either spelling.

Functions

func Do

func Do(ctx context.Context, name string, fn func(context.Context) error, kind ...Kind) error

Do runs fn inside a span, records the returned error on it and ends it. The error is returned unchanged.

func Mount

func Mount(r Router, t *Tracer) error

Mount registers the debug front end of t on r, under the path t was configured with. Mounting the tracer itself, r.Handle(path, tracer), is equivalent; this call adds the patterns each router uses to serve a subtree.

Three patterns are registered: the bare path, the trailing slash form that is the subtree on a ServeMux, and the /* wildcard that is the subtree on chi. Each router uses the ones it understands.

It returns an error when r or t is nil.

func RecordError

func RecordError(ctx context.Context, err error)

RecordError records err on the innermost span in ctx and on its trace.

if err := store.Save(ctx, u); err != nil {
	oida.RecordError(ctx, err)
	return err
}

It is Span.RecordError for code that holds a context rather than the span. A nil error, a context without a trace and an unsampled request are no-ops.

func TraceID

func TraceID(ctx context.Context) string

TraceID returns the identifier of the trace in ctx, or an empty string. It is the value of the Request-Id header for HTTP traces, which makes it the cheapest correlation key for logs.

func WithTrace

func WithTrace(ctx context.Context, t *Trace) context.Context

WithTrace returns a context carrying the trace. Spans started from the returned context, or any context derived from it, are recorded on it.

Types

type Attributes

type Attributes = model.Attributes

Attributes is a set of key/value pairs recorded on a trace or a span.

type Auth

type Auth = model.Auth

Auth evaluates the authentication options: the network allow list, the configured users, and the token verification behind the session cookie and the Authorization header. It lives in the model package so the front end can enforce it; the alias keeps it spelled oida.Auth.

func NewAuth

func NewAuth(opts Options) (*Auth, error)

NewAuth builds the authentication state out of the options, or nil when no authentication option is set: no allow list, no users and no signing secret leaves the front end open. Session mints a token from it, which is how a deployment issues one for a job that reads the dashboard API.

type Kind

type Kind = model.Kind

Kind classifies the work a span measured.

type LogEntry

type LogEntry = model.LogEntry

LogEntry is one log line recorded on a trace by Trace.Info, Trace.Error and their Span counterparts. No context is involved: a trace attributes the entry to its innermost open span, and a span uses its own id.

type Options

type Options = model.Options

Options configures telemetry behaviour, the debug front end and the middleware. It lives in the model package so the front end can read it without depending on the recorder; the alias keeps it spelled oida.Options.

func NewOptions

func NewOptions(serviceName string) Options

NewOptions returns the default options for the named service.

type Router

type Router interface {
	Handle(pattern string, h http.Handler)
}

Router is the one method chi and the standard library share: chi.Router, *chi.Mux and *http.ServeMux all register handlers with Handle, so one interface mounts the front end on either, and oida depends on neither.

type RouterFunc

type RouterFunc func(pattern string, h http.Handler)

RouterFunc adapts a registration function to Router, for a router whose own Handle does not fit:

oida.Mount(oida.RouterFunc(func(pattern string, h http.Handler) {
	r.PathPrefix(pattern).Handler(h)
}), tracer)

gorilla returns a *mux.Route from Handle and matches its paths exactly, so its dashboard is registered by prefix.

func (RouterFunc) Handle

func (f RouterFunc) Handle(pattern string, h http.Handler)

Handle implements Router.

type Sampler

type Sampler interface {
	Sample(r *http.Request) bool
}

Sampler decides whether a request is traced. The decision is taken before a trace is allocated, so rejecting a request costs one interface call. The definition is a copy of the model's; interfaces are structural, so a sampler written against either spelling works everywhere one is accepted.

Options.SampleRate covers the common case. Set Options.Sampler to decide per request on something the rate cannot express, such as a header or a route.

type Snapshot

type Snapshot = model.Snapshot

Snapshot is the complete read model of a tracer at one point in time, which is what the dashboard renders and what Tracer.Snapshot returns.

type Span

type Span = model.Span

Span is one timed operation within a trace.

func SpanFromContext

func SpanFromContext(ctx context.Context) *Span

SpanFromContext returns the innermost span in ctx, or nil.

func Start

func Start(ctx context.Context, name string, kind ...Kind) (context.Context, *Span)

Start records a span in the trace carried by ctx and returns a context carrying it. When ctx has no trace, or the trace was not sampled, it returns ctx unchanged and a nil span: every span method tolerates that.

ctx, span := oida.Start(ctx, "SELECT users", oida.KindDatabase)
defer span.End()

The kind is optional and defaults to KindInternal.

func StartAuto

func StartAuto(ctx context.Context, symbol any, kind ...Kind) (context.Context, *Span)

StartAuto is Start with the span name read from a symbol. Pass a function or a value and the package, type and function names are joined with a dot, which gives names like billing.UserStore.GetUsers without spelling them out.

ctx, span := oida.StartAuto(ctx, s.GetUsers)
defer span.End()

The name comes from reflection and the runtime symbol table, so it does not survive a stripped binary and reads oddly for anonymous functions. Use Start where either matters, or where the call is hot enough for the reflection to show up.

func StartRequest

func StartRequest(r *http.Request, name string, kind ...Kind) (*http.Request, *Span)

StartRequest is Start for code holding an *http.Request rather than a context. It returns a request carrying the span, so spans started from the returned request nest below this one.

r, span := oida.StartRequest(r, "user.Handler")
defer span.End()

When the request carries no trace it is returned unchanged along with a nil span, so the unsampled path allocates nothing.

func StartSpan

func StartSpan(ctx context.Context, name string, kind ...Kind) *Span

StartSpan records a span without deriving a context. Use it for leaf spans that will not nest.

type State

type State = model.State

State is the scoreboard state of an in-flight trace.

type Storage

type Storage interface {
	// Save retains a completed trace. The pointer is only lent for the
	// call: a driver copies what it keeps, with Clone or CloneInto, and
	// must not hold on to it.
	Save(ctx context.Context, trace *Trace) error

	// Load returns a retained trace, or ErrTraceNotFound.
	Load(ctx context.Context, id string) (Trace, error)

	// List returns retained traces newest first, at most limit of them. A limit
	// of zero or less returns everything retained.
	List(ctx context.Context, limit int) ([]Trace, error)

	// Len returns the number of retained traces.
	Len(ctx context.Context) (int, error)

	// Cap returns the retention limit, or zero when unbounded.
	Cap() int

	// Reset drops every retained trace.
	Reset(ctx context.Context) error

	// Prune drops retained traces older than maxAge. A driver with nothing
	// to prune returns nil.
	Prune(ctx context.Context, maxAge time.Duration) error

	// Restore fills the read path from what the driver persisted, so a new
	// process can list what an earlier one recorded. A driver holding nothing
	// of its own returns nil.
	Restore(ctx context.Context) error
}

Storage is what a storage driver implements to retain completed traces. Implementations must be safe for concurrent use: the tracer writes from request goroutines and reads from the debug front end at the same time.

Two drivers ship with the package, a bounded memory ring buffer and a bounded folder of JSON documents, and Configure builds either one from the environment. Set this field to retain traces somewhere else.

type Trace

type Trace = model.Trace

Trace is one recorded unit of work.

func TraceFromContext

func TraceFromContext(ctx context.Context) *Trace

TraceFromContext returns the trace in ctx, or nil.

type Tracer

type Tracer struct {
	// contains filtered or unexported fields
}

Tracer records traces into a ring buffer and serves the debug front end. The zero value is not usable; construct one with New.

func New

func New(opts Options) (*Tracer, error)

New returns a tracer built from opts. Nothing is stored in a package level variable: the tracer a request records into is the one in its context, and the tracer an entry point uses is the one handed to it.

With Options.ReadEnv set, which is what NewOptions returns, the OIDA_* environment is applied to opts first. A variable applies only where the code left the field at its default, so options set in code win over the environment, and a variable set to nothing leaves the default alone. The configuration guide lists them.

func (*Tracer) Enabled

func (t *Tracer) Enabled() bool

Enabled reports whether the tracer records traces.

func (*Tracer) Finish

func (t *Tracer) Finish(trace *Trace)

Finish completes a trace and moves it into the ring buffer. The trace keeps its recorded values, so a caller holding it may still read it; the stored copy is read back through Traces, Trace and Snapshot.

func (*Tracer) Live

func (t *Tracer) Live() []Trace

Live returns the traces currently in flight, newest first.

func (*Tracer) Middleware

func (t *Tracer) Middleware(next http.Handler) http.Handler

Middleware records every sampled request handled by next. It is compatible with chi's Use, with alice, and with any func(http.Handler) http.Handler chain:

r.Use(tracer.Middleware)

A nil tracer passes every request through, so instrumented wiring runs unchanged in a process that built none.

func (*Tracer) Observe

func (t *Tracer) Observe(ctx context.Context, name string, fn func(context.Context) error) error

Observe runs fn inside its own trace, records the returned error and completes the trace. It is what background jobs and cron ticks should use.

func (*Tracer) Options

func (t *Tracer) Options() Options

Options returns the options the tracer was built with, as a copy the caller owns. The retention driver is left out and the list and map are cloned: a reader of the configuration has no business reaching the storage behind it or rewriting what the tracer runs on.

func (*Tracer) ReportError

func (t *Tracer) ReportError(err error)

ReportError forwards a failure to Options.OnError, which is where the front end reports its render failures too. Nothing is written to stdout or stderr.

func (*Tracer) Reset

func (t *Tracer) Reset()

Reset drops every retained trace and the lifetime counters. Traces in flight are left alone and are recorded when they complete.

func (*Tracer) ServeHTTP

func (t *Tracer) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP serves the debug front end of the tracer, so a tracer mounts like any other handler:

mux := http.NewServeMux()
mux.Handle("/debug/oida/", tracer)

r := chi.NewRouter()
r.Mount("/debug/oida", tracer)

A path that does not start with Options.Path is treated as already relative, the shape http.StripPrefix delivers. A nil tracer serves 404, not a panic.

func (*Tracer) SetEnabled

func (t *Tracer) SetEnabled(enabled bool)

SetEnabled turns recording on or off at runtime. Retained traces are kept.

func (*Tracer) Snapshot

func (t *Tracer) Snapshot() Snapshot

Snapshot returns a race free copy of the tracer state. Nothing in the result aliases live state.

func (*Tracer) StartTrace

func (t *Tracer) StartTrace(ctx context.Context, name string) (context.Context, *Trace, error)

StartTrace begins a trace for work that does not arrive over HTTP. The caller must complete it with Finish.

func (*Tracer) Subscribe

func (t *Tracer) Subscribe() (<-chan struct{}, func())

Subscribe returns a channel notified whenever a trace starts or completes, and a function releasing it. The live view streams from this.

events, cancel := tracer.Subscribe()
defer cancel()
for range events {
	render(tracer.Live())
}

Notifications are coalesced, so a slow consumer cannot slow down recording.

func (*Tracer) Trace

func (t *Tracer) Trace(id string) (Trace, bool)

Trace returns the retained or in flight trace with the given ID. A retained trace is read only, the way Traces returns them; an in flight one is a copy the caller owns.

func (*Tracer) Traces

func (t *Tracer) Traces() []Trace

Traces returns the retained traces, newest first. The result is read only: its spans are the ones the front end renders, so recording into them is not a caller's to do.