o11y

README

o11y Golang SDK

A lightweight Go SDK for standardized observability, integrating OpenTelemetry (OTel) tracing with structured logging (slog) for automatic trace correlation.

Documentation

• README (this file) — project overview, infrastructure setup, SDK initialization, and the full Init options reference.
• Developer Guide — the four pillars (Tracing, Logging, Metrics, Profiling) in depth, plus per-integration sub-packages grouped by semantic-convention domain (HTTP, Databases, Messaging, Object Storage).
• Examples — runnable programs for each pillar and integration.
• Architecture Decision Records — the "why" behind key design choices.
• Semantic Conventions — pinned OTel semconv reference.

Architecture & Tech Stack

Architecture Decision Records (ADRs) explaining key design choices are in docs/adr/.

This project provides a "Context-First" observability layer for Go applications, ensuring that every log entry is automatically enriched with traceId and spanId.

• Language: Go 1.25+
• Tracing: OpenTelemetry Go SDK (OTLP/HTTP)
• Logging: Go slog with dual output — OTLP/HTTP via otelslog bridge (→ Loki) and JSON stdout (→ Alloy)
• Metrics: Prometheus pull (default :2112) or OTLP push (WithMetricsOTLPEndpoint)
• Profiling: Opt-in continuous profiling via Pyroscope (WithProfilingEndpoint)
• Infrastructure:
  • NATS: High-performance messaging
  • MongoDB: NoSQL database for persistence
  • Redis / Valkey: Cache and Redis-protocol data stores
  • Cassandra: Wide-column store via gocql (SDK-owned observers, ADR 0019)
  • Tempo: Distributed tracing backend
  • Loki: Log aggregation system
  • Pyroscope: Continuous profiling backend
  • Prometheus: Metrics storage and scraping
  • Grafana: Unified visualization for traces, logs, metrics, and profiles
  • OTel Collector: Centralized pipeline — all telemetry (traces and logs) flows through it
  • Alloy: Log collection agent and Pyroscope ingest proxy

Profiles are the one signal that bypasses the OTel Collector: applications push Pyroscope-format profiles to Alloy, which forwards them to Pyroscope.

Telemetry Flow

Traces:    App ──OTLP/HTTP──► OTel Collector ──► Tempo
Logs:      App ──OTLP/HTTP──► OTel Collector ──► Loki   (primary: full OTel Log Data Model)
           App stdout ──► Alloy ──OTLP/HTTP──► OTel Collector ──► Loki  (secondary: k8s pods via Alloy)
Metrics:   App :2112/metrics ◄──scrape── Prometheus ──► Grafana  (pull model)
Profiles:  App ──Pyroscope ingest──► Alloy ──► Pyroscope ──► Grafana

Both log paths are active simultaneously. When running go run locally (outside the cluster), only the OTLP path reaches Loki; Alloy scrapes pods exclusively inside kind. Profiles flow through Alloy's Pyroscope receiver to the Pyroscope backend; they do not go through the OTel Collector because Pyroscope ingest is not OTLP. Prometheus scraping also only works inside the cluster; locally, scrape :2112/metrics directly.

Prerequisites

Before running the infrastructure, ensure you have the following installed:

• Docker
• Go 1.25+
• kind (Kubernetes in Docker)
• kubectl

Getting Started with kind

1. Create the Cluster

kind create cluster --config kind-config.yaml

This configures a control-plane node with an extra port mapping for the OTel Collector (port 4318).

2. Deploy Infrastructure

Apply the infrastructure components using Kustomize:

# Standard deployment (public images) — monitor stack only
kubectl apply -k k8s/infrastructure/base/monitor
# Add datastores as needed for the examples you want to run, e.g.:
# kubectl apply -k k8s/infrastructure/base/components/nats

# OR: Private registry deployment (replace with your registry host)
# Note: Update internal-registry.example.com in the overlay's kustomization.yaml to your host
kubectl apply -k k8s/infrastructure/overlays/private-registry

Wait for all pods to reach the Running state.

3. Access Grafana

kubectl port-forward svc/grafana 3000:3000 -n infra

Open http://localhost:3000 (default credentials: admin / admin).

Using the SDK

This section covers initialization, the full options reference, and feature toggles. For the four pillars in depth (Tracing, Logging, Metrics, Profiling) and the per-integration sub-packages (HTTP, Databases, Messaging, Object Storage), see the Developer Guide.

Initialization

Init accepts functional options and returns an *SDK instance. No global OTel state is mutated.

import (
    "context"
    "log/slog"
    "time"

    "github.com/flywindy/o11y"
)

func main() {
    ctx := context.Background()

    obs, err := o11y.Init(ctx,
        o11y.WithServiceName("my-service"),        // required
        o11y.WithServiceVersion("1.0.0"),          // required
        o11y.WithEnvironment("production"),        // required; see canonical values below
        o11y.WithServiceNamespace("platform"),     // required; maps to k8s namespace / team
        o11y.WithOTLPEndpoint("http://localhost:4318"),
        // Optional: reduce head sampling on high-throughput producers.
        // o11y.WithSamplingRatio(0.001),
        // Optional: enable continuous profiling. In-cluster, prefer
        // "http://alloy.infra.svc.cluster.local:4040". Profiling requires
        // both the endpoint and WithProfilingEnabled(true).
        // o11y.WithProfilingEndpoint("http://localhost:4040"),
        // o11y.WithProfilingEnabled(true),
        o11y.WithLogLevel(slog.LevelInfo),
    )
    if err != nil {
        slog.ErrorContext(ctx, "failed to initialize o11y SDK", slog.Any("error", err))
        return
    }

    // Flush in-flight spans and metrics on exit (always use a timeout).
    defer func() {
        shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        if err := obs.Shutdown(shutdownCtx); err != nil {
            obs.Logger.ErrorContext(shutdownCtx, "SDK shutdown error", slog.Any("error", err))
        }
    }()
}

Available options:

Required — Init returns an error if any of these are missing:

Option	Description
WithServiceName(name)	OTel service.name resource attribute
WithServiceVersion(ver)	OTel service.version; used for canary/rollback tracking
WithEnvironment(env)	OTel deployment.environment.name; accepted: production, staging, development, testing (aliases like prod/stg are normalized)
WithServiceNamespace(ns)	OTel service.namespace; identifies the owning team/product, maps to k8s namespace

OTLP (shared by traces and logs):

Option	Default	Description
WithOTLPEndpoint(url)	http://localhost:4318	OTLP/HTTP collector endpoint for traces and logs
WithOTLPHeaders(map[string]string)	nil	Headers attached to every OTLP/HTTP request (auth tokens, multi-tenant routing)

Tracing:

Option	Default	Description
WithSamplingRatio(ratio)	unset → OTel default/env	Configure SDK-side head sampling as ParentBased(TraceIDRatioBased(ratio)); ratio must be in [0.0, 1.0]
WithTraceSampler(sampler)	unset → OTel default/env	Escape hatch for a custom sdktrace.Sampler; a non-nil sampler overrides OTEL_TRACES_SAMPLER for this SDK instance
WithBaggageAttributes(keys ...string)	nil	Materialize up to 8 application-defined W3C baggage keys onto spans and SDK log records. Keys must not collide with semconv, Resource, SDK, or slog fields
WithUserBaggage()	off	Materialize the PII-bearing semconv user.name baggage member; use ContextWithUser to set it after authentication

Application baggage is a two-sided opt-in: call ContextWithBaggageValue(ctx, key, value) where a trusted value is known, and configure WithBaggageAttributes(key) in every service that should surface it. At public ingress, exclude baggage from extraction or clear all inbound baggage before rebuilding authenticated values. Never use these keys as metric labels. See Application-Defined Baggage Attributes.

Metrics:

Option	Default	Description
WithMetricsOTLPEndpoint(url)	""	Switch metrics to OTLP push (serverless); when unset, Prometheus pull on :2112 is used
WithMetricsAddr(addr)	:2112	Prometheus /metrics scrape address
WithRuntimeMetrics(bool)	true	Collect Go runtime metrics (goroutines, GC, memory)
WithHistogramBuckets([]float64)	SLO defaults	Override HTTP latency histogram boundaries; see DefaultLatencyBuckets()
WithDisableDefaultViews()	off	Disable SDK-managed HTTP metric label allowlists and bucket views
WithMaxUniqueRoutes(n)	1000	Cap exported distinct http.route values and derive the SDK aggregation cardinality budget
WithMaxUniqueCollections(n)	200	Cap exported distinct db.collection.name values on the Cassandra and Elasticsearch client metrics, collapsing the overflow to "other" under an independent budget per integration. A Cassandra schema is DDL-fixed, so reaching this cap signals a statement shape the SDK's CQL tokenizer mis-read rather than schema growth; Elasticsearch index names often roll by date, so an "other" bucket there is the cue to opt the label out. To drop the label entirely instead, pass cassandra.WithCollectionMetricLabel(false) to cassandra.NewSession or elasticsearch.WithCollectionMetricLabel(false) to elasticsearch.NewClient
WithExtraHTTPServerAttributeKeys(keys ...string)	nil	Promote caller-controlled attribute keys (e.g. app_name, bot_name) onto the SDK-managed http.server.request.duration series. Pair with o11ygin.WithMetricAttributesFn / otelhttp equivalent to inject the values per request. Cardinality is the caller's responsibility — use enumerable, bounded keyspaces
WithExemplars(bool)	true	Enable OpenMetrics negotiation on /metrics so per-bucket exemplars carry trace_id / span_id to Grafana / Tempo. Set false only as a temporary mitigation when migrating dashboards that hardcode integer histogram boundaries (le="1" → le="1.0" under OpenMetrics)

Logging:

Option	Default	Description
WithLogLevel(level)	slog.LevelInfo	Minimum log level

Profiling (opt-in via endpoint):

Option	Default	Description
WithProfilingEndpoint(url)	""	Pyroscope-compatible ingest endpoint; empty means profiling never starts
WithProfilingAuthHeaders(map[string]string)	nil	Headers attached to every profile push (Grafana Cloud Profiles auth, X-Scope-OrgID, etc.)

Per-pillar feature toggles (progressive rollout):

Option	Default	Description
WithTraceEnabled(bool)	true	When false, use a no-op TracerProvider; W3C headers are still parsed and forwarded
WithMetricsEnabled(bool)	true	When false, use a no-op MeterProvider; no Prometheus server is started
WithLogEnabled(bool)	true	When false, write logs to stdout only; no OTLP log provider is started
WithProfilingEnabled(bool)	false	Opt-in. When true and WithProfilingEndpoint is set, the SDK starts the Pyroscope profiler and installs the trace-to-profile bridge

Migration note (pre-1.0 API change) — DefaultLatencyBuckets is now a function (o11y.DefaultLatencyBuckets() returning a fresh copy) rather than a package-level slice variable, so callers cannot accidentally mutate the package defaults. DefaultMetricsAddr is now a const (was var); SDK.TracerProvider() now returns trace.TracerProvider, and SDK.MeterProvider() now returns metric.MeterProvider. Use WithMetricsAddr(":9090") to override the metrics listener. See CHANGELOG.md for the migration recipe.

Feature Toggles (Progressive Rollout)

Each observability pillar — Trace, Metrics, Log, Profiling — can be controlled independently so teams can adopt the new SDK incrementally without breaking existing dashboards or pipelines.

obs, err := o11y.Init(ctx,
    // ... required options ...
    o11y.WithTraceEnabled(false),     // keep existing trace logic, skip new SDK
    o11y.WithMetricsEnabled(false),   // keep ginprom /metrics; no new Prometheus server
    o11y.WithLogEnabled(false),       // stdout only; no OTLP log export
    o11y.WithProfilingEndpoint("http://alloy.infra.svc.cluster.local:4040"),
    o11y.WithProfilingEnabled(true),  // opt-in: profiling defaults to off
)

When disabled, each pillar is gracefully stubbed out:

Pillar	Disabled behaviour
Trace	No-op TracerProvider; W3C traceparent/tracestate headers are still parsed and forwarded to downstream services
Metrics	No-op MeterProvider; the Prometheus HTTP server is not started; existing ginprom dashboards are unaffected
Log	obs.Logger writes to stdout only (JSON); no OTLP collector connection is attempted
Profiling	No Pyroscope profiler is started; the trace-to-profile pyroscope.profile.id span attribute is not added. Trace/log/metric pillars are untouched

WithTraceEnabled(false) returns a no-op TracerProvider; integration code that is already installed may still execute a no-op wrapper path. NATS callers can supply the SDK's resolved toggle as their connection-local tracing default:

conn, err := o11ynats.ConnectWithOptions(
    ctx,
    natsURL,
    obs.TracerProvider(),
    obs.Propagator,
    o11ynats.WithTracingEnabled(obs.Toggles.Trace),
    o11ynats.WithNATSOptions(natsOpts...),
)

Since otel-nats v0.8.0, the NATS-specific precedence is relay > upstream environment > connection option > upstream default. The option above selects the direct/native path only when no feature-flag relay and no overriding OTEL_NATS_TRACING_ENABLED value are present. A relay-capable process keeps the instrumented path available and evaluates flags per operation even while the effective flag is false. This SDK does not configure a relay; adopting one is a separate application/deployment decision.

Above that whole ladder sits an upstream master switch, OTEL_INSTRUMENTATION_GO_TRACING_ENABLED, which is ANDed with the result. It defaults to enabled, so leaving it unset changes nothing — but setting it to a falsy value turns off NATS tracing (and with it header propagation and baggage restoration) no matter what the option, the module variable or the relay say. Both upstream variables are strict tri-state since v0.8.0: only 1/true/yes/on and 0/false/no/off are accepted, and any other value — including the empty string an unexpanded ${VAR} produces — fails the connection with an error rather than being ignored. Audit deployment configuration for these two names before upgrading.

Environment-variable control (useful for staged rollouts without deploys):

Env var	Default
O11Y_TRACE_ENABLED	true
O11Y_METRICS_ENABLED	true
O11Y_LOG_ENABLED	true
O11Y_PROFILING_ENABLED	false

Accepted values: 1/t/true/TRUE (truthy) and 0/f/false/FALSE (falsy). Any other value (e.g. "yes", "on") emits a startup WARN log and falls back to the SDK default.

Precedence: code option > env var > SDK default.
An explicit WithTraceEnabled(true) always wins over O11Y_TRACE_ENABLED=false.

Profiling is opt-in and doubly gated: it requires both WithProfilingEnabled(true) (or O11Y_PROFILING_ENABLED=true) and a non-empty WithProfilingEndpoint. Either condition alone is insufficient, so misconfiguration cannot accidentally start the profiler. The SDK emits a startup WARN when only one of the two is set so the misconfig is noticed. sdk.Toggles.Profiling reports whether the SDK actually started a profiler.

Runtime introspection — sdk.Toggles reports what is active:

if !obs.Toggles.Metrics {
    obs.Logger.Warn("metrics pillar disabled; Prometheus server not started")
}
if !obs.Toggles.Profiling {
    obs.Logger.Warn("profiling inactive; toggle off or no Pyroscope endpoint")
}

Creating Spans

Use obs.Tracer(name) to obtain a named tracer. No global OTel tracer provider is required.

tracer := obs.Tracer("my-service")

ctx, span := tracer.Start(ctx, "parent-operation")
defer span.End()

// Child span — inherits the trace from ctx
ctx, child := tracer.Start(ctx, "child-operation")
defer child.End()

obs.Logger.InfoContext(ctx, "child work done")

If you need to wire the SDK's provider into the global OTel state (e.g. for third-party libraries that call otel.Tracer()):

import "go.opentelemetry.io/otel"

otel.SetTracerProvider(obs.TracerProvider())
otel.SetTextMapPropagator(obs.Propagator)

For everything else — structured logging with trace correlation, user identity attributes, trace sampling, continuous profiling, and the NATS / MongoDB / Redis / Elasticsearch / HTTP / Resty / MinIO / gin sub-packages — see the Developer Guide.

Background work & context lifecycle

Work that outlives the request — background goroutines, post-response writes, fire-and-forget — must not use the request context (c.Request.Context() or *gin.Context). It is canceled the moment the handler returns, which aborts the work mid-flight (context canceled, and for databases connection reset by peer). The failure is timing-dependent, so it tends to pass locally and only surface under production latency/concurrency.

Use obsctx to keep the trace context but drop the cancelation:

import "github.com/flywindy/o11y/obsctx"

// in a gin handler, after responding:
obsctx.Go(c.Request.Context(), 5*time.Second, func(ctx context.Context) {
    _ = repo.WriteAudit(ctx, event) // stays in the same trace; not canceled with the request
})

See examples/background and ADR 0024.

Examples

Runnable programs live in examples/, organized like the Developer Guide: the four pillars first (basic spans + logs, metrics, profiling), then integrations grouped by semantic-convention domain (HTTP: gin, Resty; Databases: MongoDB, Redis; Messaging: NATS Core, JetStream, browser WebSocket; Object Storage: MinIO). See examples/README.md for the prerequisites (port-forwards) and the go run command for each one.

Core Principles

1. Context-First: Always propagate context.Context — trace information flows through context only.
2. Zero Global State: No init() side effects, no global logger or tracer provider variables. See ADR 0003.
3. Correlation: Every log record includes traceId and spanId when a span is active — as JSON fields on stdout and as OTel Log Data Model fields in Loki. See ADR 0001.
4. Errors: Use slog.ErrorContext(ctx, ...) with structured attributes; never panic for recoverable errors.
5. Semconv v1.39.0: All instrument names, attribute keys, and types conform to OTel Semantic Conventions v1.39.0. See docs/semconv.md.

Acknowledgements

• github.com/akira-core/instrumentation-go/otel-nats — provides the underlying NATS Core + JetStream tracing semantics used by the nats/ wrapper (module path renamed from Marz32onE to akira-core upstream in v0.6.0). Verified at v0.9.1 not to mutate global OTel providers/propagators and to import semconv v1.39.0. When explicitly configured through OTEL_INSTRUMENTATION_GO_FLAGS_ENDPOINT, v0.8.0 may install a named OpenFeature provider; o11y does not set that variable. See ADR 0004 for the integration decision and audit discipline.
• go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/v2/mongo/otelmongo provides MongoDB command spans and db.client.operation.duration metrics through the mongo facade; the SDK owns MongoDB connection-pool metrics. See ADR 0014 and ADR 0021.
• github.com/redis/go-redis/v9 — is the Redis/Valkey client wrapped by the SDK-owned redis/ instrumentation. The wrapper does not call redisotel; see ADR 0013.
• github.com/elastic/go-elasticsearch/v8 — ships first-party OpenTelemetry tracing in its shared transport; the elasticsearch/ facade wires the SDK TracerProvider into it (search body opt-in) without touching OTel globals, and records an SDK-owned db.client.operation.duration on the SDK MeterProvider since the upstream is trace-only. See ADR 0020 and ADR 0027.
• github.com/gocql/gocql — is the Cassandra driver instrumented by the SDK-owned cassandra/ observers. The otelgocql contrib package was removed upstream, so the SDK owns the observer code; see ADR 0019.
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp — provides the underlying HTTP server/client instrumentation used by the http/ facade. See ADR 0009.
• github.com/grafana/pyroscope-go and github.com/grafana/otel-profiling-go provide the Pyroscope profiler and trace-to-profile bridge. See ADR 0012.
• go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin — provides the underlying gin instrumentation used by the gin/ facade. See ADR 0010.
• github.com/go-resty/resty/v2 is the HTTP client wrapped by the SDK-owned resty instrumentation. The wrapper does not use otelhttp for Resty; it owns span lifecycle in Resty hooks so retry attempts stay observable. See ADR 0011.

AI Collaboration

This project uses AGENTS.md to store AI-assisted development context and project-specific rules. CLAUDE.md and GEMINI.md are symlinks pointing to that file. If using an AI assistant, refer to AGENTS.md for project patterns.

Documentation

Overview

Package o11y is the top-level entry point for the SDK. It exposes Init for constructing a configured *SDK that bundles trace, metric, log, and optional profiling providers together with the W3C TraceContext+Baggage propagator and a dual-output slog logger.

The SDK never mutates global OpenTelemetry state; callers wire the returned providers into their application explicitly (e.g. otel.SetTracerProvider).

See ADR 0001 (log format strategy), ADR 0002 (metrics strategy), and ADR 0007 (OTLP authentication) for the design rationale.

Index

• Constants
• func ContextWithBaggageValue(ctx context.Context, key, value string) (context.Context, error)
• func ContextWithUser(ctx context.Context, name string) (context.Context, error)
• func ContextWithoutBaggageValues(ctx context.Context, keys ...string) context.Context
• func DefaultLatencyBuckets() []float64
• func SetUser(ctx context.Context, name string)
• func UserName(name string) slog.Attr
• type Config
• type FeatureToggles
• type Option
  • func WithBaggageAttributes(keys ...string) Option
  • func WithDisableDefaultViews() Option
  • func WithEnvironment(env string) Option
  • func WithExemplars(enabled bool) Option
  • func WithExtraHTTPServerAttributeKeys(keys ...string) Option
  • func WithHistogramBuckets(buckets []float64) Option
  • func WithLogEnabled(enabled bool) Option
  • func WithLogLevel(level slog.Level) Option
  • func WithMaxUniqueCollections(n int) Option
  • func WithMaxUniqueRoutes(n int) Option
  • func WithMetricsAddr(addr string) Option
  • func WithMetricsEnabled(enabled bool) Option
  • func WithMetricsOTLPEndpoint(endpoint string) Option
  • func WithOTLPEndpoint(endpoint string) Option
  • func WithOTLPHeaders(headers map[string]string) Option
  • func WithProfilingAuthHeaders(headers map[string]string) Option
  • func WithProfilingEnabled(enabled bool) Option
  • func WithProfilingEndpoint(endpoint string) Option
  • func WithRuntimeMetrics(enabled bool) Option
  • func WithSamplingRatio(ratio float64) Option
  • func WithServiceName(name string) Option
  • func WithServiceNamespace(namespace string) Option
  • func WithServiceVersion(version string) Option
  • func WithTraceEnabled(enabled bool) Option
  • func WithTraceSampler(sampler sdktrace.Sampler) Option
  • func WithUserBaggage() Option
• type SDK
  • func Init(ctx context.Context, opts ...Option) (*SDK, error)
  • func (s *SDK) Meter(name string) metric.Meter
  • func (s *SDK) MeterProvider() metric.MeterProvider
  • func (s *SDK) Shutdown(ctx context.Context) error
  • func (s *SDK) Tracer(name string) oteltrace.Tracer
  • func (s *SDK) TracerProvider() oteltrace.TracerProvider

Constants

const (
	// UserNameKey is the semantic-convention key for the acting user's login name.
	UserNameKey = baggageattrs.UserNameKey

	// MaxBaggageValueBytes is the maximum value size accepted by SDK baggage setters.
	MaxBaggageValueBytes = baggageattrs.MaxBaggageValueBytes

	// MaxBaggageKeyBytes is the maximum key size accepted by SDK baggage setters.
	MaxBaggageKeyBytes = baggageattrs.MaxBaggageKeyBytes
)

const DefaultMaxUniqueCollections = 200

DefaultMaxUniqueCollections is the default export-boundary cap for distinct db.collection.name values on the Cassandra client metrics.

A Cassandra schema's table count is fixed by DDL and is normally in the tens, so this is not a budget the label is expected to approach — it is a backstop against a statement shape the SDK's CQL tokenizer mis-reads, which would otherwise turn a bounded label into an unbounded one. It is set well below DefaultMaxUniqueRoutes because a schema is a much smaller keyspace than a service's URL space.

const DefaultMaxUniqueRoutes = 1000

DefaultMaxUniqueRoutes is the default export-boundary cap for distinct http.route values. Additional SDK cardinality limits protect in-process aggregation memory before export.

const DefaultMetricsAddr = ":2112"

DefaultMetricsAddr is the default listen address for the built-in Prometheus /metrics HTTP server.

const MaxBaggageAttributeKeys = 8

MaxBaggageAttributeKeys bounds the application-defined baggage keys one SDK instance materializes. user.name has a separate opt-in and does not count.

Functions

func ContextWithBaggageValue

func ContextWithBaggageValue(ctx context.Context, key, value string) (context.Context, error)

ContextWithBaggageValue returns a child context carrying key=value as a W3C baggage member. It validates the W3C token, key and value lengths, SDK and semantic-convention reservations, and the serialized W3C baggage budget. It leaves ctx unchanged on failure. user.name is reserved for ContextWithUser. Keys are limited to MaxBaggageKeyBytes, values to MaxBaggageValueBytes, and the resulting baggage to 64 wire-serializable members and 8192 encoded bytes.

An empty value is a no-op after key validation; it does not remove an existing member. Use ContextWithoutBaggageValues for selective removal, or baggage.ContextWithoutBaggage at a public ingress. Set values only after authentication/authorization and strip baggage before untrusted egress.

func ContextWithUser

func ContextWithUser(ctx context.Context, name string) (context.Context, error)

ContextWithUser returns a child context that carries the acting user's login name as the OpenTelemetry baggage member user.name.

This is the source-side opt-in for ADR 0016 automatic user identity propagation. Because the SDK propagator already includes W3C Baggage, the value can travel across HTTP and NATS boundaries. Use this only after authenticating the user, and strip baggage before egress to untrusted third parties. Services must also opt in with WithUserBaggage to materialize the baggage value onto their own spans and logs. Empty names return the original context without adding baggage. Values over MaxBaggageValueBytes or additions that would exceed the serialized W3C baggage budget return an error and leave the original context unchanged.

func ContextWithoutBaggageValues

func ContextWithoutBaggageValues(ctx context.Context, keys ...string) context.Context

ContextWithoutBaggageValues returns a child context with the named baggage members removed while preserving every other member. It is intended for internal trust boundaries. At public ingress, clear all baggage instead so unknown keys cannot pass through to newer downstream services.

func DefaultLatencyBuckets

func DefaultLatencyBuckets() []float64

DefaultLatencyBuckets returns a fresh copy of the SDK's default histogram boundaries. It returns a copy so that callers who keep a reference cannot accidentally mutate the package-level defaults.

func SetUser

func SetUser(ctx context.Context, name string)

SetUser records the acting user's login name on the current span as the OpenTelemetry semantic convention attribute user.name.

SetUser is an explicit, in-process helper: it does not add the field to log records, does not store the value in baggage, and does not propagate it across service boundaries. Use UserName alongside SetUser when the same username should also be present on a slog record. Empty names are ignored.

func UserName

func UserName(name string) slog.Attr

UserName returns a slog attribute for the acting user's login name using the OpenTelemetry semantic convention key user.name.

UserName is an explicit log helper: it only affects the log record where the returned attribute is supplied. It does not record the username on spans and does not propagate it across service boundaries. Empty names return an empty slog attribute.

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

Config defines the configuration for the o11y SDK.

type FeatureToggles

type FeatureToggles struct {
	Trace     bool
	Metrics   bool
	Log       bool
	Profiling bool
}

FeatureToggles reports which observability pillars were enabled at Init time. Use sdk.Toggles to inspect active state at runtime, e.g. for health-check endpoints or to conditionally log a startup warning when a pillar is off.

Profiling reflects the combined state of WithProfilingEnabled and whether a non-empty Pyroscope endpoint was configured: it is true only when the SDK actually started a profiler.

type Option

type Option func(*Config)

Option is a functional option for configuring the o11y SDK.

func WithBaggageAttributes

func WithBaggageAttributes(keys ...string) Option

WithBaggageAttributes enables materialization of application-defined W3C baggage members onto spans and SDK log records. This option does not create baggage; use ContextWithBaggageValue after validating the source value. Calls accumulate and de-duplicate, and the first MaxBaggageAttributeKeys valid application keys are used.

Empty, non-token, overlong, and reserved keys are dropped with a startup warning. Reservations include SDK and slog fields, the complete pinned semantic-convention catalog and parameterized namespaces, and user.name, whose PII contract requires WithUserBaggage. Init fails if an otherwise valid key collides with this process's effective Resource attributes, including OTEL_RESOURCE_ATTRIBUTES. Never use a materialized key as a metric label.

func WithDisableDefaultViews

func WithDisableDefaultViews() Option

WithDisableDefaultViews disables SDK-managed HTTP metric views.

func WithEnvironment

func WithEnvironment(env string) Option

WithEnvironment sets the deployment environment (e.g., "production", "staging").

func WithExemplars

func WithExemplars(enabled bool) Option

WithExemplars controls whether the Prometheus pull `/metrics` handler content-negotiates the OpenMetrics exposition format. OpenMetrics is the only format the Prometheus exporter renders per-bucket exemplars in, so this option also governs whether trace-to-metric linkage in Grafana / Tempo works for SDK-managed and caller-defined histograms. Default: true.

Set to false only as a temporary mitigation when migrating a service whose existing PromQL queries, recording rules, or alert rules hardcode integer histogram bucket boundaries (e.g. `le="1"`, `le="5"`, `le="10"`). The OpenMetrics format emits those as `le="1.0"`, `le="5.0"`, `le="10.0"` — the underlying bucket boundary is identical (`float64`) but the rendered label string differs, so queries matching the integer form silently stop matching after rollout. Aggregate queries such as `histogram_quantile(...)` are unaffected. Update the dependent queries, then remove this option.

Has no effect on the OTLP push path (WithMetricsOTLPEndpoint): exemplars travel in the OTLP proto and are not impacted by Prometheus text-format negotiation.

func WithExtraHTTPServerAttributeKeys

func WithExtraHTTPServerAttributeKeys(keys ...string) Option

WithExtraHTTPServerAttributeKeys extends the SDK-managed allow-list on the http.server.request.duration metric view. By default that view keeps only http.request.method, http.route, and http.response.status_code to bound cardinality; any other attributes attached to the record (for example via o11ygin.WithMetricAttributesFn or otelhttp's WithMetricAttributesFn) are dropped from the exported series and end up as exemplar labels, where the OpenMetrics 128-rune cap quickly trips. Use this option to promote a small set of caller-controlled keys (e.g. "app_name", "bot_name") onto the series itself so they participate in PromQL aggregations.

Cardinality is the caller's responsibility: every distinct value combination multiplies the existing route×method×status series count. Prefer keys whose value space is enumerable and small (tens, not thousands).

Keys are checked against the otelprom Prometheus label-name normalization (non-alphanumeric → '_', runs collapsed, leading digits prefixed with "key_"). The SDK drops — with a startup WARN log — any key that, after normalization:

• matches a built-in label the SDK already exports (the view-allowed HTTP semconv keys, the four resource constants, and the three otelprom scope labels otel_scope_name / _version / _schema_url), or
• matches another caller-supplied key from this or a prior WithExtraHTTPServerAttributeKeys call (e.g. "app.name" and "app_name" both normalize to "app_name"), or
• normalizes to the empty string.

Accepting either form of collision would silently merge two attribute values into a single Prometheus label, corrupting PromQL grouping for that dimension.

Calls accumulate. Has no effect when WithDisableDefaultViews is set, because no SDK-managed view exists to extend.

func WithHistogramBuckets

func WithHistogramBuckets(buckets []float64) Option

WithHistogramBuckets overrides the histogram boundaries applied to HTTP server and client latency histograms. Defaults to DefaultLatencyBuckets; override only when your service has a genuinely different latency profile. Changing these from the package default makes cross-service P99 comparisons inconsistent.

func WithLogEnabled

func WithLogEnabled(enabled bool) Option

WithLogEnabled controls whether the SDK exports logs via OTLP to the OTel Collector. When false, slog records are written to stdout only; no OTLP log provider is started and no collector connection is attempted. Default: true (env var O11Y_LOG_ENABLED overrides the built-in default).

func WithLogLevel

func WithLogLevel(level slog.Level) Option

WithLogLevel returns an Option that sets the minimum logging level for the SDK.

func WithMaxUniqueCollections

func WithMaxUniqueCollections(n int) Option

WithMaxUniqueCollections sets the distinct db.collection.name export cap for the Cassandra client metrics (db.client.operation.duration and cassandra.query.attempts) and the Elasticsearch client metric (db.client.operation.duration, where the collection is the index). Values <= 0 use DefaultMaxUniqueCollections. Each integration has its own budget of n values, so an Elasticsearch overflow never evicts Cassandra tables or vice versa.

Values beyond the cap are collapsed to the literal label "other" at the export boundary, the same mechanism WithMaxUniqueRoutes applies to http.route. Because a Cassandra schema's tables are DDL-fixed, reaching the cap there normally means the SDK's CQL tokenizer mis-read a statement shape rather than that the schema genuinely grew; an "other" bucket on the Cassandra metrics is worth investigating rather than raising the cap. Elasticsearch index names commonly roll by date, so an "other" bucket there is the expected signal to opt the label out (below) rather than a defect.

Callers who would rather not carry the label at all should pass cassandra.WithCollectionMetricLabel(false) to NewSession or elasticsearch.WithCollectionMetricLabel(false) to NewClient instead — this cap bounds the label, it does not remove it.

func WithMaxUniqueRoutes

func WithMaxUniqueRoutes(n int) Option

WithMaxUniqueRoutes sets the distinct http.route export cap. Values <= 0 use DefaultMaxUniqueRoutes. The SDK also derives an in-process aggregation cardinality budget from this value to guard against unbounded attribute sets.

func WithMetricsAddr

func WithMetricsAddr(addr string) Option

WithMetricsAddr returns an Option that sets the metrics HTTP server listen address to the provided addr. If not set, the metrics server defaults to DefaultMetricsAddr (":2112").

func WithMetricsEnabled

func WithMetricsEnabled(enabled bool) Option

WithMetricsEnabled controls whether the SDK initialises a real MeterProvider. When false, no Prometheus HTTP server is started and no OTLP metrics are exported. All instrumentation that accepts a MeterProvider receives a no-op provider, preserving compile-time compatibility with zero runtime cost. Default: true (env var O11Y_METRICS_ENABLED overrides the built-in default).

func WithMetricsOTLPEndpoint

func WithMetricsOTLPEndpoint(endpoint string) Option

WithMetricsOTLPEndpoint switches the metrics exporter from Prometheus pull to OTLP push. When set, the /metrics HTTP server is not started and metrics are exported via OTLP/HTTP to the given endpoint. Use this for serverless environments (Lambda, Cloud Run) where exposing a scrape port is not possible. When unset, the default Prometheus pull model is used.

Example: o11y.WithMetricsOTLPEndpoint("http://collector:4318")

func WithOTLPEndpoint

func WithOTLPEndpoint(endpoint string) Option

WithOTLPEndpoint sets the OTLP/HTTP collector endpoint used for traces and logs. The endpoint must be reachable by the SDK's process (no proxy is configured by this package).

Production note: prefer https:// in production deployments. The default http://localhost:4318 is intended for local development against an OTel Collector running on the same host. When sending telemetry across a network boundary, use TLS — observability traffic carries trace IDs, hostnames, error messages and stack traces that should not be exposed in plaintext.

If the endpoint requires authentication (Grafana Cloud, Honeycomb, NewRelic, Datadog, ...), pair this option with WithOTLPHeaders to attach the API token / Bearer header to every OTLP request.

func WithOTLPHeaders

func WithOTLPHeaders(headers map[string]string) Option

WithOTLPHeaders attaches custom HTTP headers to every OTLP/HTTP request emitted by the SDK (traces, logs, and OTLP metrics push). Typical use cases:

• Authentication against managed observability backends, e.g. {"Authorization": "Bearer <token>"} or {"X-Honeycomb-Team": "<api-key>"}.
• Multi-tenant routing on a shared Collector, e.g. {"X-Scope-OrgID": "<tenant>"}.

Calling WithOTLPHeaders multiple times merges into the same map; later calls overwrite earlier values for the same header key. Header values are not logged.

func WithProfilingAuthHeaders

func WithProfilingAuthHeaders(headers map[string]string) Option

WithProfilingAuthHeaders attaches custom HTTP headers to every Pyroscope profile push. Use this for Grafana Cloud Profiles, Basic auth via an Authorization header, or multi-tenant routing such as X-Scope-OrgID.

Calling WithProfilingAuthHeaders multiple times merges into the same map; later calls overwrite earlier values for the same header key. Header values are not logged.

func WithProfilingEnabled

func WithProfilingEnabled(enabled bool) Option

WithProfilingEnabled controls whether the SDK starts the Pyroscope profiler and wraps the TracerProvider with the trace-to-profile bridge. Profiling is opt-in: it requires both this toggle to be on AND a non-empty endpoint set via WithProfilingEndpoint. Either condition alone is insufficient. This lets operators stage a profiling rollout (or roll it back) without removing the endpoint from deployment manifests.

Default: false (env var O11Y_PROFILING_ENABLED overrides the built-in default). Unlike trace/metrics/log, profiling defaults off because it is an additional fourth signal that must be explicitly enabled.

func WithProfilingEndpoint

func WithProfilingEndpoint(endpoint string) Option

WithProfilingEndpoint sets the Pyroscope-compatible ingest endpoint. Examples:

• "http://alloy.infra.svc.cluster.local:4040" for Grafana Alloy
• "http://pyroscope.infra.svc.cluster.local:4040" for direct Pyroscope ingest

When empty, profiling is fully disabled: no profiler goroutines are started, no pprof globals are touched, and trace spans are not annotated with pyroscope.profile.id. Default: empty.

func WithRuntimeMetrics

func WithRuntimeMetrics(enabled bool) Option

WithRuntimeMetrics toggles collection of Go runtime metrics (goroutines, GC, memory, etc.) via OTel runtime instrumentation. Defaults to true.

func WithSamplingRatio

func WithSamplingRatio(ratio float64) Option

WithSamplingRatio configures head sampling for root traces using ParentBased(TraceIDRatioBased(ratio)). Use this on high-throughput producer services to reduce in-process span allocation, batch-processor pressure, and downstream trace volume while preserving whole-trace consistency through context propagation.

The ratio must be between 0.0 and 1.0 inclusive; Init returns an error for out-of-range, NaN, or infinite values instead of relying on OTel's native clamping. When this option is set, it overrides OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG for this SDK instance. When unset, those OTel env vars remain active.

func WithServiceName

func WithServiceName(name string) Option

WithServiceName sets the service name for trace resource attributes.

func WithServiceNamespace

func WithServiceNamespace(namespace string) Option

WithServiceNamespace sets the service.namespace resource attribute (OTel semconv). It is required: Init returns an error when empty. The value identifies the owning team or product unit and maps naturally to the Kubernetes namespace when services are namespaced by product. It becomes a constant Prometheus label (service_namespace="...") on every series and appears on all three observability signals (traces, logs, metrics).

func WithServiceVersion

func WithServiceVersion(version string) Option

WithServiceVersion sets the service version (e.g. "1.4.2") for trace resource attributes. Used in OTel as service.version and is especially useful for canary deployments and version-based trace filtering.

func WithTraceEnabled

func WithTraceEnabled(enabled bool) Option

WithTraceEnabled controls whether the SDK initialises a real TracerProvider and exports spans via OTLP. When false, Init returns a no-op TracerProvider; the W3C TraceContext propagator still parses and forwards trace headers so downstream services are unaffected. Default: true (env var O11Y_TRACE_ENABLED overrides the built-in default).

func WithTraceSampler

func WithTraceSampler(sampler sdktrace.Sampler) Option

WithTraceSampler configures an explicit OpenTelemetry sampler for the SDK's TracerProvider. It is an escape hatch for custom samplers not expressible via WithSamplingRatio or the OTEL_TRACES_SAMPLER environment variable.

Passing nil leaves sampling unset, preserving the OTel environment/default sampler path. A non-nil sampler overrides OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG for this SDK instance.

func WithUserBaggage

func WithUserBaggage() Option

WithUserBaggage enables ADR 0016 opt-in materialization of whitelisted user identity baggage onto this service's spans and log records.

This option does not create baggage by itself. Use ContextWithUser after authentication to put user.name into the context; WithUserBaggage then copies that whitelisted value onto spans at start time and onto slog records emitted by this SDK's logger. The feature is off by default because user.name is PII and propagated baggage can cross service boundaries. user.name is tracked separately from application keys and does not consume MaxBaggageAttributeKeys.

type SDK

type SDK struct {
	// Logger writes structured log records to two destinations:
	//   • stdout       – JSON with service.name, traceId, and spanId fields
	//                    (for local development and container log collection via Fluentd)
	//   • OTel Collector – OTLP/HTTP → Loki (full OTel Log Data Model; service
	//                    identity comes from the shared Resource, not per-record attrs)
	// When a span is active in the context, traceId and spanId are included
	// automatically in both destinations. They stay at the record's top level
	// even on a logger derived with Logger.WithGroup, so log-to-trace queries
	// keyed on the top-level field keep matching; the group still nests that
	// logger's own attributes as usual.
	// When WithLogEnabled(false) is set, only the stdout destination is active.
	Logger *slog.Logger

	// Propagator is the W3C TraceContext + Baggage composite propagator.
	// Pass it to nats.Inject / nats.Extract for distributed tracing over NATS.
	// The propagator is always set (even when trace is disabled) so that
	// incoming trace headers are still parsed and forwarded to downstream services.
	Propagator propagation.TextMapPropagator

	// Toggles reports which observability pillars were enabled at Init time.
	Toggles FeatureToggles
	// contains filtered or unexported fields
}

SDK holds the initialized observability providers. It does not mutate any global state; callers wire it however they like, e.g. slog.SetDefault(obs.Logger) or otel.SetTracerProvider(obs.TracerProvider()).

func Init

func Init(ctx context.Context, opts ...Option) (*SDK, error)

Init initializes and returns a configured *SDK for the calling service.

The following options are required; Init returns an error if any are missing or invalid:

• WithServiceName — identifies the service
• WithServiceVersion — used for canary / rollback tracking
• WithEnvironment — must be one of: production, staging, development, testing (common aliases such as "prod" and "stg" are normalized automatically)
• WithServiceNamespace — identifies the owning team / k8s namespace

On success the SDK contains a tracer provider, meter provider (Prometheus scrape or OTLP push), logger provider (stdout JSON + OTLP/HTTP → Loki), and an ordered shutdown list. Init does not set global OpenTelemetry state.

func (*SDK) Meter

func (s *SDK) Meter(name string) metric.Meter

Meter returns a named meter from the SDK's MeterProvider for custom instrumentation. HTTP server and client instrumentation are provided by the http.NewServerHandler and http.NewTransport facades, which accept the SDK's MeterProvider directly.

func (*SDK) MeterProvider

func (s *SDK) MeterProvider() metric.MeterProvider

MeterProvider returns the SDK's meter provider interface. Use this when wiring SDK-produced metrics into instrumentation libraries that accept an OTel MeterProvider directly.

func (*SDK) Shutdown

func (s *SDK) Shutdown(ctx context.Context) error

Shutdown gracefully flushes and shuts down all registered SDK components. Each component is attempted even if a previous one fails; all errors are logged and returned joined. Always call with a context that has a timeout to cap the flush wait.

Shutdown is idempotent: subsequent calls return the same joined error without rerunning any closer. Callers may safely register Shutdown in multiple defer chains (for example, both in main and in a signal handler) without risking double-shutdown of underlying exporters.

func (*SDK) Tracer

func (s *SDK) Tracer(name string) oteltrace.Tracer

Tracer returns a named tracer from the SDK's TracerProvider.

func (*SDK) TracerProvider

func (s *SDK) TracerProvider() oteltrace.TracerProvider

TracerProvider returns the SDK's tracer provider interface. Use this to wire the SDK's provider as the global OTel tracer provider if needed, e.g. otel.SetTracerProvider(sdk.TracerProvider()).