o11y

README

o11y Golang SDK

A lightweight Go SDK for standardized observability, integrating OpenTelemetry (OTel) tracing with structured logging (slog) for automatic trace correlation.

Architecture & Tech Stack

Architecture Decision Records (ADRs) explaining key design choices are in docs/adr/.

This project provides a "Context-First" observability layer for Go applications, ensuring that every log entry is automatically enriched with traceId and spanId.

• Language: Go 1.25+
• Tracing: OpenTelemetry Go SDK (OTLP/HTTP)
• Logging: Go slog with dual output — OTLP/HTTP via otelslog bridge (→ Loki) and JSON stdout (→ Alloy)
• Metrics: Prometheus pull (default :2112) or OTLP push (WithMetricsOTLPEndpoint)
• Profiling: Opt-in continuous profiling via Pyroscope (WithProfilingEndpoint)
• Infrastructure:
  • NATS: High-performance messaging
  • MongoDB: NoSQL database for persistence
  • Tempo: Distributed tracing backend
  • Loki: Log aggregation system
  • Pyroscope: Continuous profiling backend
  • Prometheus: Metrics storage and scraping
  • Grafana: Unified visualization for traces, logs, metrics, and profiles
  • OTel Collector: Centralized pipeline — all telemetry (traces and logs) flows through it
  • Alloy: Log collection agent and Pyroscope ingest proxy

Profiles are the one signal that bypasses the OTel Collector: applications push Pyroscope-format profiles to Alloy, which forwards them to Pyroscope.

Telemetry Flow

Traces:    App ──OTLP/HTTP──► OTel Collector ──► Tempo
Logs:      App ──OTLP/HTTP──► OTel Collector ──► Loki   (primary: full OTel Log Data Model)
           App stdout ──► Alloy ──OTLP/HTTP──► OTel Collector ──► Loki  (secondary: k8s pods via Alloy)
Metrics:   App :2112/metrics ◄──scrape── Prometheus ──► Grafana  (pull model)
Profiles:  App ──Pyroscope ingest──► Alloy ──► Pyroscope ──► Grafana

Both log paths are active simultaneously. When running go run locally (outside the cluster), only the OTLP path reaches Loki; Alloy scrapes pods exclusively inside kind. Profiles flow through Alloy's Pyroscope receiver to the Pyroscope backend; they do not go through the OTel Collector because Pyroscope ingest is not OTLP. Prometheus scraping also only works inside the cluster; locally, scrape :2112/metrics directly.

Prerequisites

Before running the infrastructure, ensure you have the following installed:

• Docker
• Go 1.25+
• kind (Kubernetes in Docker)
• kubectl

Getting Started with kind

1. Create the Cluster

kind create cluster --config kind-config.yaml

This configures a control-plane node with an extra port mapping for the OTel Collector (port 4318).

2. Deploy Infrastructure

Apply the infrastructure components using Kustomize:

# Standard deployment (public images)
kubectl apply -k k8s/infrastructure/base

# OR: Private registry deployment (replace with your registry host)
# Note: Update internal-registry.example.com in the overlay's kustomization.yaml to your host
kubectl apply -k k8s/infrastructure/overlays/private-registry

Wait for all pods to reach the Running state.

3. Access Grafana

kubectl port-forward svc/grafana 3000:3000 -n infra

Open http://localhost:3000 (default credentials: admin / admin).

Using the SDK

Initialization

Init accepts functional options and returns an *SDK instance. No global OTel state is mutated.

import (
    "context"
    "log/slog"
    "time"

    "github.com/flywindy/o11y"
)

func main() {
    ctx := context.Background()

    obs, err := o11y.Init(ctx,
        o11y.WithServiceName("my-service"),        // required
        o11y.WithServiceVersion("1.0.0"),          // required
        o11y.WithEnvironment("production"),        // required; see canonical values below
        o11y.WithServiceNamespace("platform"),     // required; maps to k8s namespace / team
        o11y.WithOTLPEndpoint("http://localhost:4318"),
        // Optional: enable continuous profiling. In-cluster, prefer
        // "http://alloy.infra.svc.cluster.local:4040". Profiling requires
        // both the endpoint and WithProfilingEnabled(true).
        // o11y.WithProfilingEndpoint("http://localhost:4040"),
        // o11y.WithProfilingEnabled(true),
        o11y.WithLogLevel(slog.LevelInfo),
    )
    if err != nil {
        slog.ErrorContext(ctx, "failed to initialize o11y SDK", slog.Any("error", err))
        return
    }

    // Flush in-flight spans and metrics on exit (always use a timeout).
    defer func() {
        shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        if err := obs.Shutdown(shutdownCtx); err != nil {
            obs.Logger.ErrorContext(shutdownCtx, "SDK shutdown error", slog.Any("error", err))
        }
    }()
}

Available options:

Required — Init returns an error if any of these are missing:

Option	Description
WithServiceName(name)	OTel service.name resource attribute
WithServiceVersion(ver)	OTel service.version; used for canary/rollback tracking
WithEnvironment(env)	OTel deployment.environment.name; accepted: production, staging, development, testing (aliases like prod/stg are normalized)
WithServiceNamespace(ns)	OTel service.namespace; identifies the owning team/product, maps to k8s namespace

OTLP (shared by traces and logs):

Option	Default	Description
WithOTLPEndpoint(url)	http://localhost:4318	OTLP/HTTP collector endpoint for traces and logs
WithOTLPHeaders(map[string]string)	nil	Headers attached to every OTLP/HTTP request (auth tokens, multi-tenant routing)

Metrics:

Option	Default	Description
WithMetricsOTLPEndpoint(url)	""	Switch metrics to OTLP push (serverless); when unset, Prometheus pull on :2112 is used
WithMetricsAddr(addr)	:2112	Prometheus /metrics scrape address
WithRuntimeMetrics(bool)	true	Collect Go runtime metrics (goroutines, GC, memory)
WithHistogramBuckets([]float64)	SLO defaults	Override HTTP latency histogram boundaries; see DefaultLatencyBuckets()
WithDisableDefaultViews()	off	Disable SDK-managed HTTP metric label allowlists and bucket views
WithMaxUniqueRoutes(n)	1000	Cap exported distinct http.route values and derive the SDK aggregation cardinality budget
WithExtraHTTPServerAttributeKeys(keys ...string)	nil	Promote caller-controlled attribute keys (e.g. app_name, bot_name) onto the SDK-managed http.server.request.duration series. Pair with o11ygin.WithMetricAttributesFn / otelhttp equivalent to inject the values per request. Cardinality is the caller's responsibility — use enumerable, bounded keyspaces

Logging:

Option	Default	Description
WithLogLevel(level)	slog.LevelInfo	Minimum log level

Profiling (opt-in via endpoint):

Option	Default	Description
WithProfilingEndpoint(url)	""	Pyroscope-compatible ingest endpoint; empty means profiling never starts
WithProfilingAuthHeaders(map[string]string)	nil	Headers attached to every profile push (Grafana Cloud Profiles auth, X-Scope-OrgID, etc.)

Per-pillar feature toggles (progressive rollout):

Option	Default	Description
WithTraceEnabled(bool)	true	When false, use a no-op TracerProvider; W3C headers are still parsed and forwarded
WithMetricsEnabled(bool)	true	When false, use a no-op MeterProvider; no Prometheus server is started
WithLogEnabled(bool)	true	When false, write logs to stdout only; no OTLP log provider is started
WithProfilingEnabled(bool)	false	Opt-in. When true and WithProfilingEndpoint is set, the SDK starts the Pyroscope profiler and installs the trace-to-profile bridge

Migration note (pre-1.0 API change) — DefaultLatencyBuckets is now a function (o11y.DefaultLatencyBuckets() returning a fresh copy) rather than a package-level slice variable, so callers cannot accidentally mutate the package defaults. DefaultMetricsAddr is now a const (was var); SDK.TracerProvider() now returns trace.TracerProvider, and SDK.MeterProvider() now returns metric.MeterProvider. Use WithMetricsAddr(":9090") to override the metrics listener. See CHANGELOG.md for the migration recipe.

Feature Toggles (Progressive Rollout)

Each observability pillar — Trace, Metrics, Log, Profiling — can be controlled independently so teams can adopt the new SDK incrementally without breaking existing dashboards or pipelines.

obs, err := o11y.Init(ctx,
    // ... required options ...
    o11y.WithTraceEnabled(false),     // keep existing trace logic, skip new SDK
    o11y.WithMetricsEnabled(false),   // keep ginprom /metrics; no new Prometheus server
    o11y.WithLogEnabled(false),       // stdout only; no OTLP log export
    o11y.WithProfilingEndpoint("http://alloy.infra.svc.cluster.local:4040"),
    o11y.WithProfilingEnabled(true),  // opt-in: profiling defaults to off
)

When disabled, each pillar is gracefully stubbed out:

Pillar	Disabled behaviour
Trace	No-op TracerProvider; W3C traceparent/tracestate headers are still parsed and forwarded to downstream services
Metrics	No-op MeterProvider; the Prometheus HTTP server is not started; existing ginprom dashboards are unaffected
Log	obs.Logger writes to stdout only (JSON); no OTLP collector connection is attempted
Profiling	No Pyroscope profiler is started; the trace-to-profile pyroscope.profile.id span attribute is not added. Trace/log/metric pillars are untouched

Environment-variable control (useful for staged rollouts without deploys):

Env var	Default
O11Y_TRACE_ENABLED	true
O11Y_METRICS_ENABLED	true
O11Y_LOG_ENABLED	true
O11Y_PROFILING_ENABLED	false

Accepted values: 1/t/true/TRUE (truthy) and 0/f/false/FALSE (falsy). Any other value (e.g. "yes", "on") emits a startup WARN log and falls back to the SDK default.

Precedence: code option > env var > SDK default.
An explicit WithTraceEnabled(true) always wins over O11Y_TRACE_ENABLED=false.

Profiling is opt-in and doubly gated: it requires both WithProfilingEnabled(true) (or O11Y_PROFILING_ENABLED=true) and a non-empty WithProfilingEndpoint. Either condition alone is insufficient, so misconfiguration cannot accidentally start the profiler. The SDK emits a startup WARN when only one of the two is set so the misconfig is noticed. sdk.Toggles.Profiling reports whether the SDK actually started a profiler.

Runtime introspection — sdk.Toggles reports what is active:

if !obs.Toggles.Metrics {
    obs.Logger.Warn("metrics pillar disabled; Prometheus server not started")
}
if !obs.Toggles.Profiling {
    obs.Logger.Warn("profiling inactive; toggle off or no Pyroscope endpoint")
}

Structured Logging with Trace Correlation

Use obs.Logger instead of the global slog package. Every log record is written to two destinations automatically:

• OTLP → Loki: Full OTel Log Data Model. service.name and deployment.environment live in the OTel Resource (not per-record attributes). traceId, spanId, and trace_flags are extracted from the context by the otelslog bridge.
• stdout (JSON): Human-readable output for local development. Includes service.name, environment, traceId, and spanId as flat JSON fields.
• When WithLogEnabled(false) is set, only the stdout destination is active.

// Without a span — no trace fields in either destination
obs.Logger.Info("service started")

// With an active span — trace context included automatically
ctx, span := obs.Tracer("my-tracer").Start(ctx, "my-operation")
defer span.End()

obs.Logger.InfoContext(ctx, "processing request", slog.String("user_id", "42"))
// stdout: {"time":"...","level":"INFO","msg":"processing request","service.name":"my-service","traceId":"4bf92f...","spanId":"00f067...","user_id":"42"}
// Loki:   OTel Log Record — Body="processing request", TraceId=4bf92f..., SpanId=00f067..., Attributes={user_id: "42"}, Resource={service.name: "my-service", ...}

Logging Guidelines

The dual-output logger forwards every record to two backends. Treat both as shared, queryable infrastructure — anything you log is searchable by every engineer with cluster access.

• Never log secrets: API tokens, session cookies, signed URLs, full Authorization headers, internal IPs, JWTs, raw OAuth state, encryption keys. Redact before passing to slog. The WithOTLPHeaders option intentionally does not log header values for the same reason.
• Hash or truncate user identifiers: prefer slog.String("user_id", hash(uid)) over the raw email/phone. traceId already lets you correlate a single user's request across logs without storing PII.
• Never log raw request bodies: a malicious client can plant \n{"level":"INFO",...} inside a body field and inject a synthetic log line into your stdout JSON pipeline. If you must record body shape, log only the field count or a schema hash.
• Use *Context variants: Logger.InfoContext(ctx, ...) (not Info(...)) so that traceId and spanId are populated. A log without trace correlation is operationally a needle in a haystack.
• Pre-validate attribute keys: slog.String(userInput, ...) lets the attacker control the log's field name. Use a fixed key and put untrusted data in the value.
• Watch attribute size: slog.Any happily serializes arbitrary structs. A 10 MB struct logged at 1 kHz overruns both stdout and the OTLP exporter's batch queue. Cap or summarise large payloads before logging.

Creating Spans

Use obs.Tracer(name) to obtain a named tracer. No global OTel tracer provider is required.

tracer := obs.Tracer("my-service")

ctx, span := tracer.Start(ctx, "parent-operation")
defer span.End()

// Child span — inherits the trace from ctx
ctx, child := tracer.Start(ctx, "child-operation")
defer child.End()

obs.Logger.InfoContext(ctx, "child work done")

If you need to wire the SDK's provider into the global OTel state (e.g. for third-party libraries that call otel.Tracer()):

import "go.opentelemetry.io/otel"

otel.SetTracerProvider(obs.TracerProvider())
otel.SetTextMapPropagator(obs.Propagator)

Continuous Profiling

Profiling is opt-in and doubly gated: it requires both a non-empty WithProfilingEndpoint AND WithProfilingEnabled(true) (or O11Y_PROFILING_ENABLED=true). Either alone is insufficient. In the provided Kubernetes stack, applications should send profiles to Alloy:

obs, err := o11y.Init(ctx,
    o11y.WithServiceName("orders-api"),
    o11y.WithServiceVersion("1.0.0"),
    o11y.WithEnvironment("production"),
    o11y.WithServiceNamespace("platform"),
    o11y.WithOTLPEndpoint("http://otel-collector.infra.svc.cluster.local:4318"),
    o11y.WithProfilingEndpoint("http://alloy.infra.svc.cluster.local:4040"),
    o11y.WithProfilingEnabled(true),
)

For local development, port-forward Alloy and use WithProfilingEndpoint("http://localhost:4040"). Use WithProfilingAuthHeaders when the Pyroscope endpoint requires auth or tenant routing headers. Header values are copied defensively and are not logged.

When profiling is enabled, the SDK wraps its tracer provider with the Grafana span profiling bridge. Root spans receive a pyroscope.profile.id attribute, and Pyroscope samples are labeled so Grafana can open CPU profiles from Tempo. The link is statistical: short spans, especially below the CPU sampling interval, can legitimately show an empty profile.

Important caveats:

• Trace-to-profile navigation is CPU-profile-only. Service-level profiles also include allocation and in-use memory profiles.
• By default, only local root spans are labeled by the bridge.
• Go pprof labels apply to the current goroutine. Work started in a new goroutine is captured in the service-level profile, but it is not linked to the span unless the application propagates pprof labels explicitly.

Distributed Tracing over NATS

Use obs.Propagator together with the nats sub-package to propagate trace context across NATS messages.

import (
    o11ynats "github.com/flywindy/o11y/nats"
    gonats "github.com/nats-io/nats.go"
)

conn, err := o11ynats.Connect(ctx, natsURL, obs.TracerProvider(), obs.Propagator)

// Publisher: trace context is injected into message headers automatically.
if err := conn.Publish(ctx, "orders.created", payload); err != nil {
    obs.Logger.ErrorContext(ctx, "publish failed", slog.Any("error", err))
}

// Subscriber: ctx in the handler already carries the publisher's trace.
// Subscribe returns (*nats.Subscription, error) — capture both: the error
// surfaces invalid input (empty subject, nil handler, cancelled ctx) and the
// Subscription handle is what you call Unsubscribe()/Drain() on at shutdown.
sub, err := conn.Subscribe(ctx, "orders.created", func(ctx context.Context, msg *gonats.Msg) {
    ctx, span := obs.Tracer("consumer").Start(ctx, "orders.created")
    defer span.End()
    obs.Logger.InfoContext(ctx, "order received") // traceId and spanId injected automatically
})
if err != nil {
    obs.Logger.ErrorContext(ctx, "subscribe failed", slog.Any("error", err))
    return
}
defer func() { _ = sub.Drain() }() // gracefully drain on shutdown

MongoDB

Use the mongo sub-package to wire MongoDB tracing to the SDK-owned TracerProvider and Propagator. Do not import the upstream otel-mongo/v2 package directly from application code.

import o11ymongo "github.com/flywindy/o11y/mongo"

client, err := o11ymongo.Connect(ctx, mongoURI, obs.TracerProvider(), obs.Propagator)
if err != nil {
    obs.Logger.ErrorContext(ctx, "MongoDB connect failed", slog.Any("error", err))
    return
}
defer func() {
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    _ = client.Disconnect(shutdownCtx)
}()

collection := client.Database("app").Collection("orders")
_, err = collection.InsertOne(ctx, bson.M{"_id": "order-123", "status": "created"})

MongoDB command spans are gated by the upstream instrumentation flags:

export OTEL_INSTRUMENTATION_GO_TRACING_ENABLED=true
export OTEL_MONGO_TRACING_ENABLED=true

Document trace propagation is disabled by default because it writes an _oteltrace field into persisted documents. Enable it only for asynchronous patterns such as change streams or outbox processors that need to restore trace context from MongoDB documents:

client, err := o11ymongo.Connect(ctx, mongoURI, obs.TracerProvider(), obs.Propagator,
    o11ymongo.WithDocumentTracePropagation(true),
)

Prometheus Metrics

By default the SDK exposes a /metrics endpoint on :2112 for Prometheus to scrape. Every series carries service_namespace, service_name, service_version, and deployment_environment_name as constant labels.

curl http://localhost:2112/metrics   # inspect raw output

HTTP handler instrumentation is provided by the github.com/flywindy/o11y/http package:

import o11yhttp "github.com/flywindy/o11y/http"

mux := http.NewServeMux()
mux.HandleFunc("GET /api/orders/{id}", handleOrder)

// Wrap the mux. The SDK passes TracerProvider, MeterProvider, and Propagator
// explicitly, so otelhttp never reads OpenTelemetry globals.
handler := o11yhttp.NewServerHandler(
    mux,
    obs.TracerProvider(),
    obs.MeterProvider(),
    obs.Propagator,
)

For Go 1.22+ http.ServeMux, route patterns become bounded span names such as GET /api/orders/{id} and bounded http.route metric labels. For routers such as chi or echo, use their route pattern as an otelhttp label or span-name formatter at the router edge; keep raw URL paths out of metric labels. WithMaxUniqueRoutes rewrites excess exported server routes to http_route="other" while the OTel SDK's own cardinality limit protects in-process aggregators from attacker-controlled attribute sets. If the separate SDK guard trips, metrics are preserved under otel_metric_overflow="true" with route detail intentionally dropped.

Using with gin

Use the gin sub-package to wire gin's OTel middleware to the SDK-owned TracerProvider, MeterProvider, and Propagator. Register the returned chain before gin.Recovery() so panics recovered by gin still produce complete HTTP status attributes and metrics.

import (
    "errors"
    "net/http"

    o11ygin "github.com/flywindy/o11y/gin"
    "github.com/gin-gonic/gin"
)

router := gin.New()
router.Use(o11ygin.Middleware(
    "orders-api",
    obs.TracerProvider(),
    obs.MeterProvider(),
    obs.Propagator,
)...)
router.Use(gin.Recovery())

router.GET("/orders/:id", func(c *gin.Context) {
    c.JSON(http.StatusOK, gin.H{"status": "ok"})
})
router.GET("/fail", func(c *gin.Context) {
    err := errors.New("simulated failure")
    c.AbortWithError(http.StatusInternalServerError, err).SetType(gin.ErrorTypePublic)
})

ErrorRecorder adds typed gin.error.type span events for errors pushed via c.Error / c.AbortWithError. The metric label set remains governed by the SDK's HTTP metric views and does not include gin error types.

Exemplars are enabled automatically (OTel SDK default trace-based filter). When Prometheus is deployed with --enable-feature=exemplar-storage (included in k8s/infrastructure/base/prometheus.yaml), Grafana can navigate from a histogram bucket directly to the correlated trace in Tempo. The measurement context must contain an active sampled span; exemplar trace IDs are stored as exemplar metadata (trace_id / span_id), not as metric labels, so they do not create high-cardinality time series.

Kubernetes pods must opt in to scraping with the annotation:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "2112"   # optional; 2112 is the default

Running the Examples

Before running any example, port-forward the required services from the kind cluster:

kubectl port-forward -n infra svc/otel-collector 4318:4318  # OTel traces and logs
kubectl port-forward -n infra svc/nats           4222:4222  # NATS connection
kubectl port-forward -n infra svc/grafana        3000:3000  # Grafana UI
kubectl port-forward -n infra svc/prometheus     9090:9090  # Prometheus UI
kubectl port-forward -n infra svc/alloy          4040:4040  # Pyroscope ingest for local app profiling

Basic (spans + logs)

go run examples/basic/main.go

NATS Core (two terminals)

# Terminal 1 — start subscriber first
go run examples/nats-core/subscriber/main.go

# Terminal 2 — publisher sends a message every 3 seconds
go run examples/nats-core/publisher/main.go

JetStream (two terminals; requires JetStream-enabled NATS server)

# Terminal 1 — publisher creates the stream and publishes
go run examples/jetstream/publisher/main.go

# Terminal 2 — subscriber attaches a durable consumer and processes messages
go run examples/jetstream/subscriber/main.go

Metrics (otelhttp facade + OTLP push)

go run examples/metrics/main.go

The example starts an HTTP server on :8080 and generates synthetic traffic every 500 ms. Metrics flow via OTLP/HTTP to the OTel Collector, which forwards them to Prometheus via remote write. It uses the same localhost:4318 NodePort as traces and logs, so no extra metrics scrape port is needed for this example. Histogram buckets include exemplars linking each measurement to its trace.

Open Grafana at http://localhost:3000 and navigate to:

• Explore → Tempo — producer and consumer spans linked across services
• Explore → Loki — structured log entries with correlated traceId and spanId
• Explore → Prometheus — http_server_request_duration_seconds; click an exemplar dot to jump to the linked trace in Tempo
• Dashboards → Observability → Metrics Correlation — HTTP latency metrics with a data link that opens the matching metrics-example logs in Loki

Gin

go run examples/gin/main.go
curl http://localhost:8080/ok
curl http://localhost:8080/fail

The example registers o11ygin.Middleware(...) before gin.Recovery() and demonstrates typed gin.error.type span events from c.AbortWithError.

Profiling

go run examples/profiling/main.go

The example starts a sampled root span every two seconds and burns CPU long enough for Pyroscope to capture useful samples. It sends profiles to PYROSCOPE_ENDPOINT (default http://localhost:4040) and traces/logs/OTLP metrics to OTLP_ENDPOINT (default http://localhost:4318). Keep the Alloy and Grafana port-forwards from the setup block running while the example runs.

MongoDB

Run a local MongoDB instance or port-forward one to localhost:27017, then enable the upstream tracing gates before running the example:

export MONGODB_URI=mongodb://localhost:27017
export OTEL_INSTRUMENTATION_GO_TRACING_ENABLED=true
export OTEL_MONGO_TRACING_ENABLED=true
go run examples/mongodb/main.go

To demonstrate _oteltrace document propagation, opt in explicitly:

export O11Y_MONGO_DOCUMENT_TRACE_PROPAGATION=true
go run examples/mongodb/main.go

Core Principles

1. Context-First: Always propagate context.Context — trace information flows through context only.
2. Zero Global State: No init() side effects, no global logger or tracer provider variables. See ADR 0003.
3. Correlation: Every log record includes traceId and spanId when a span is active — as JSON fields on stdout and as OTel Log Data Model fields in Loki. See ADR 0001.
4. Errors: Use slog.ErrorContext(ctx, ...) with structured attributes; never panic for recoverable errors.
5. Semconv v1.39.0: All instrument names, attribute keys, and types conform to OTel Semantic Conventions v1.39.0. See docs/semconv.md.

Acknowledgements

• github.com/Marz32onE/instrumentation-go/otel-nats — provides the underlying NATS Core + JetStream tracing semantics used by the nats/ wrapper. Verified at v0.2.11 not to mutate OTel globals and to import semconv v1.39.0. See ADR 0004 for the integration decision and audit discipline.
• github.com/Marz32onE/instrumentation-go/otel-mongo/v2 — provides the underlying MongoDB driver v2 tracing semantics used by the mongo/ wrapper. Verified at the otel-mongo/v2/v0.2.11 tag not to mutate OTel globals, with _oteltrace document propagation disabled by default through the o11y wrapper. See ADR 0005.
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp — provides the underlying HTTP server/client instrumentation used by the http/ facade. See ADR 0009.
• github.com/grafana/pyroscope-go and github.com/grafana/otel-profiling-go provide the Pyroscope profiler and trace-to-profile bridge. See ADR 0012.
• go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin — provides the underlying gin instrumentation used by the gin/ facade. See ADR 0010.

AI Collaboration

This project uses AGENTS.md to store AI-assisted development context and project-specific rules. CLAUDE.md and GEMINI.md are symlinks pointing to that file. If using an AI assistant, refer to AGENTS.md for project patterns.

Documentation

Overview

Package o11y is the top-level entry point for the SDK. It exposes Init for constructing a configured *SDK that bundles trace, metric, log, and optional profiling providers together with the W3C TraceContext+Baggage propagator and a dual-output slog logger.

The SDK never mutates global OpenTelemetry state; callers wire the returned providers into their application explicitly (e.g. otel.SetTracerProvider).

See ADR 0001 (log format strategy), ADR 0002 (metrics strategy), and ADR 0007 (OTLP authentication) for the design rationale.

Index

• Constants
• func DefaultLatencyBuckets() []float64
• type Config
• type FeatureToggles
• type Option
  • func WithDisableDefaultViews() Option
  • func WithEnvironment(env string) Option
  • func WithExtraHTTPServerAttributeKeys(keys ...string) Option
  • func WithHistogramBuckets(buckets []float64) Option
  • func WithLogEnabled(enabled bool) Option
  • func WithLogLevel(level slog.Level) Option
  • func WithMaxUniqueRoutes(n int) Option
  • func WithMetricsAddr(addr string) Option
  • func WithMetricsEnabled(enabled bool) Option
  • func WithMetricsOTLPEndpoint(endpoint string) Option
  • func WithOTLPEndpoint(endpoint string) Option
  • func WithOTLPHeaders(headers map[string]string) Option
  • func WithProfilingAuthHeaders(headers map[string]string) Option
  • func WithProfilingEnabled(enabled bool) Option
  • func WithProfilingEndpoint(endpoint string) Option
  • func WithRuntimeMetrics(enabled bool) Option
  • func WithServiceName(name string) Option
  • func WithServiceNamespace(namespace string) Option
  • func WithServiceVersion(version string) Option
  • func WithTraceEnabled(enabled bool) Option
• type SDK
  • func Init(ctx context.Context, opts ...Option) (*SDK, error)
  • func (s *SDK) Meter(name string) metric.Meter
  • func (s *SDK) MeterProvider() metric.MeterProvider
  • func (s *SDK) Shutdown(ctx context.Context) error
  • func (s *SDK) Tracer(name string) oteltrace.Tracer
  • func (s *SDK) TracerProvider() oteltrace.TracerProvider

Constants

const DefaultMaxUniqueRoutes = 1000

DefaultMaxUniqueRoutes is the default export-boundary cap for distinct http.route values. Additional SDK cardinality limits protect in-process aggregation memory before export.

const DefaultMetricsAddr = ":2112"

DefaultMetricsAddr is the default listen address for the built-in Prometheus /metrics HTTP server.

Functions

func DefaultLatencyBuckets

func DefaultLatencyBuckets() []float64

DefaultLatencyBuckets returns a fresh copy of the SDK's default histogram boundaries. It returns a copy so that callers who keep a reference cannot accidentally mutate the package-level defaults.

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

Config defines the configuration for the o11y SDK.

type FeatureToggles

type FeatureToggles struct {
	Trace     bool
	Metrics   bool
	Log       bool
	Profiling bool
}

FeatureToggles reports which observability pillars were enabled at Init time. Use sdk.Toggles to inspect active state at runtime, e.g. for health-check endpoints or to conditionally log a startup warning when a pillar is off.

Profiling reflects the combined state of WithProfilingEnabled and whether a non-empty Pyroscope endpoint was configured: it is true only when the SDK actually started a profiler.

type Option

type Option func(*Config)

Option is a functional option for configuring the o11y SDK.

func WithDisableDefaultViews

func WithDisableDefaultViews() Option

WithDisableDefaultViews disables SDK-managed HTTP metric views.

func WithEnvironment

func WithEnvironment(env string) Option

WithEnvironment sets the deployment environment (e.g., "production", "staging").

func WithExtraHTTPServerAttributeKeys

func WithExtraHTTPServerAttributeKeys(keys ...string) Option

WithExtraHTTPServerAttributeKeys extends the SDK-managed allow-list on the http.server.request.duration metric view. By default that view keeps only http.request.method, http.route, and http.response.status_code to bound cardinality; any other attributes attached to the record (for example via o11ygin.WithMetricAttributesFn or otelhttp's WithMetricAttributesFn) are dropped from the exported series and end up as exemplar labels, where the OpenMetrics 128-rune cap quickly trips. Use this option to promote a small set of caller-controlled keys (e.g. "app_name", "bot_name") onto the series itself so they participate in PromQL aggregations.

Cardinality is the caller's responsibility: every distinct value combination multiplies the existing route×method×status series count. Prefer keys whose value space is enumerable and small (tens, not thousands).

Keys are checked against the otelprom Prometheus label-name normalization (non-alphanumeric → '_', runs collapsed, leading digits prefixed with "key_"). The SDK drops — with a startup WARN log — any key that, after normalization:

• matches a built-in label the SDK already exports (the view-allowed HTTP semconv keys, the four resource constants, and the three otelprom scope labels otel_scope_name / _version / _schema_url), or
• matches another caller-supplied key from this or a prior WithExtraHTTPServerAttributeKeys call (e.g. "app.name" and "app_name" both normalize to "app_name"), or
• normalizes to the empty string.

Accepting either form of collision would silently merge two attribute values into a single Prometheus label, corrupting PromQL grouping for that dimension.

Calls accumulate. Has no effect when WithDisableDefaultViews is set, because no SDK-managed view exists to extend.

func WithHistogramBuckets

func WithHistogramBuckets(buckets []float64) Option

WithHistogramBuckets overrides the histogram boundaries applied to HTTP server and client latency histograms. Defaults to DefaultLatencyBuckets; override only when your service has a genuinely different latency profile. Changing these from the package default makes cross-service P99 comparisons inconsistent.

func WithLogEnabled

func WithLogEnabled(enabled bool) Option

WithLogEnabled controls whether the SDK exports logs via OTLP to the OTel Collector. When false, slog records are written to stdout only; no OTLP log provider is started and no collector connection is attempted. Default: true (env var O11Y_LOG_ENABLED overrides the built-in default).

func WithLogLevel

func WithLogLevel(level slog.Level) Option

WithLogLevel returns an Option that sets the minimum logging level for the SDK.

func WithMaxUniqueRoutes

func WithMaxUniqueRoutes(n int) Option

WithMaxUniqueRoutes sets the distinct http.route export cap. Values <= 0 use DefaultMaxUniqueRoutes. The SDK also derives an in-process aggregation cardinality budget from this value to guard against unbounded attribute sets.

func WithMetricsAddr

func WithMetricsAddr(addr string) Option

WithMetricsAddr returns an Option that sets the metrics HTTP server listen address to the provided addr. If not set, the metrics server defaults to DefaultMetricsAddr (":2112").

func WithMetricsEnabled

func WithMetricsEnabled(enabled bool) Option

WithMetricsEnabled controls whether the SDK initialises a real MeterProvider. When false, no Prometheus HTTP server is started and no OTLP metrics are exported. All instrumentation that accepts a MeterProvider receives a no-op provider, preserving compile-time compatibility with zero runtime cost. Default: true (env var O11Y_METRICS_ENABLED overrides the built-in default).

func WithMetricsOTLPEndpoint

func WithMetricsOTLPEndpoint(endpoint string) Option

WithMetricsOTLPEndpoint switches the metrics exporter from Prometheus pull to OTLP push. When set, the /metrics HTTP server is not started and metrics are exported via OTLP/HTTP to the given endpoint. Use this for serverless environments (Lambda, Cloud Run) where exposing a scrape port is not possible. When unset, the default Prometheus pull model is used.

Example: o11y.WithMetricsOTLPEndpoint("http://collector:4318")

func WithOTLPEndpoint

func WithOTLPEndpoint(endpoint string) Option

WithOTLPEndpoint sets the OTLP/HTTP collector endpoint used for traces and logs. The endpoint must be reachable by the SDK's process (no proxy is configured by this package).

Production note: prefer https:// in production deployments. The default http://localhost:4318 is intended for local development against an OTel Collector running on the same host. When sending telemetry across a network boundary, use TLS — observability traffic carries trace IDs, hostnames, error messages and stack traces that should not be exposed in plaintext.

If the endpoint requires authentication (Grafana Cloud, Honeycomb, NewRelic, Datadog, ...), pair this option with WithOTLPHeaders to attach the API token / Bearer header to every OTLP request.

func WithOTLPHeaders

func WithOTLPHeaders(headers map[string]string) Option

WithOTLPHeaders attaches custom HTTP headers to every OTLP/HTTP request emitted by the SDK (traces, logs, and OTLP metrics push). Typical use cases:

• Authentication against managed observability backends, e.g. {"Authorization": "Bearer <token>"} or {"X-Honeycomb-Team": "<api-key>"}.
• Multi-tenant routing on a shared Collector, e.g. {"X-Scope-OrgID": "<tenant>"}.

Calling WithOTLPHeaders multiple times merges into the same map; later calls overwrite earlier values for the same header key. Header values are not logged.

func WithProfilingAuthHeaders

func WithProfilingAuthHeaders(headers map[string]string) Option

WithProfilingAuthHeaders attaches custom HTTP headers to every Pyroscope profile push. Use this for Grafana Cloud Profiles, Basic auth via an Authorization header, or multi-tenant routing such as X-Scope-OrgID.

Calling WithProfilingAuthHeaders multiple times merges into the same map; later calls overwrite earlier values for the same header key. Header values are not logged.

func WithProfilingEnabled

func WithProfilingEnabled(enabled bool) Option

WithProfilingEnabled controls whether the SDK starts the Pyroscope profiler and wraps the TracerProvider with the trace-to-profile bridge. Profiling is opt-in: it requires both this toggle to be on AND a non-empty endpoint set via WithProfilingEndpoint. Either condition alone is insufficient. This lets operators stage a profiling rollout (or roll it back) without removing the endpoint from deployment manifests.

Default: false (env var O11Y_PROFILING_ENABLED overrides the built-in default). Unlike trace/metrics/log, profiling defaults off because it is an additional fourth signal that must be explicitly enabled.

func WithProfilingEndpoint

func WithProfilingEndpoint(endpoint string) Option

WithProfilingEndpoint sets the Pyroscope-compatible ingest endpoint. Examples:

• "http://alloy.infra.svc.cluster.local:4040" for Grafana Alloy
• "http://pyroscope.infra.svc.cluster.local:4040" for direct Pyroscope ingest

When empty, profiling is fully disabled: no profiler goroutines are started, no pprof globals are touched, and trace spans are not annotated with pyroscope.profile.id. Default: empty.

func WithRuntimeMetrics

func WithRuntimeMetrics(enabled bool) Option

WithRuntimeMetrics toggles collection of Go runtime metrics (goroutines, GC, memory, etc.) via OTel runtime instrumentation. Defaults to true.

func WithServiceName

func WithServiceName(name string) Option

WithServiceName sets the service name for trace resource attributes.

func WithServiceNamespace

func WithServiceNamespace(namespace string) Option

WithServiceNamespace sets the service.namespace resource attribute (OTel semconv). It is required: Init returns an error when empty. The value identifies the owning team or product unit and maps naturally to the Kubernetes namespace when services are namespaced by product. It becomes a constant Prometheus label (service_namespace="...") on every series and appears on all three observability signals (traces, logs, metrics).

func WithServiceVersion

func WithServiceVersion(version string) Option

WithServiceVersion sets the service version (e.g. "1.4.2") for trace resource attributes. Used in OTel as service.version and is especially useful for canary deployments and version-based trace filtering.

func WithTraceEnabled

func WithTraceEnabled(enabled bool) Option

WithTraceEnabled controls whether the SDK initialises a real TracerProvider and exports spans via OTLP. When false, Init returns a no-op TracerProvider; the W3C TraceContext propagator still parses and forwards trace headers so downstream services are unaffected. Default: true (env var O11Y_TRACE_ENABLED overrides the built-in default).

type SDK

type SDK struct {
	// Logger writes structured log records to two destinations:
	//   • stdout       – JSON with service.name, traceId, and spanId fields
	//                    (for local development and container log collection via Fluentd)
	//   • OTel Collector – OTLP/HTTP → Loki (full OTel Log Data Model; service
	//                    identity comes from the shared Resource, not per-record attrs)
	// When a span is active in the context, traceId and spanId are included
	// automatically in both destinations.
	// When WithLogEnabled(false) is set, only the stdout destination is active.
	Logger *slog.Logger

	// Propagator is the W3C TraceContext + Baggage composite propagator.
	// Pass it to nats.Inject / nats.Extract for distributed tracing over NATS.
	// The propagator is always set (even when trace is disabled) so that
	// incoming trace headers are still parsed and forwarded to downstream services.
	Propagator propagation.TextMapPropagator

	// Toggles reports which observability pillars were enabled at Init time.
	Toggles FeatureToggles
	// contains filtered or unexported fields
}

SDK holds the initialized observability providers. It does not mutate any global state; callers wire it however they like, e.g. slog.SetDefault(obs.Logger) or otel.SetTracerProvider(obs.TracerProvider()).

func Init

func Init(ctx context.Context, opts ...Option) (*SDK, error)

Init initializes and returns a configured *SDK for the calling service.

The following options are required; Init returns an error if any are missing or invalid:

• WithServiceName — identifies the service
• WithServiceVersion — used for canary / rollback tracking
• WithEnvironment — must be one of: production, staging, development, testing (common aliases such as "prod" and "stg" are normalized automatically)
• WithServiceNamespace — identifies the owning team / k8s namespace

On success the SDK contains a tracer provider, meter provider (Prometheus scrape or OTLP push), logger provider (stdout JSON + OTLP/HTTP → Loki), and an ordered shutdown list. Init does not set global OpenTelemetry state.

func (*SDK) Meter

func (s *SDK) Meter(name string) metric.Meter

Meter returns a named meter from the SDK's MeterProvider for custom instrumentation. HTTP server and client instrumentation are provided by the http.NewServerHandler and http.NewTransport facades, which accept the SDK's MeterProvider directly.

func (*SDK) MeterProvider

func (s *SDK) MeterProvider() metric.MeterProvider

MeterProvider returns the SDK's meter provider interface. Use this when wiring SDK-produced metrics into instrumentation libraries that accept an OTel MeterProvider directly.

func (*SDK) Shutdown

func (s *SDK) Shutdown(ctx context.Context) error

Shutdown gracefully flushes and shuts down all registered SDK components. Each component is attempted even if a previous one fails; all errors are logged and returned joined. Always call with a context that has a timeout to cap the flush wait.

Shutdown is idempotent: subsequent calls return the same joined error without rerunning any closer. Callers may safely register Shutdown in multiple defer chains (for example, both in main and in a signal handler) without risking double-shutdown of underlying exporters.

func (*SDK) Tracer

func (s *SDK) Tracer(name string) oteltrace.Tracer

Tracer returns a named tracer from the SDK's TracerProvider.

func (*SDK) TracerProvider

func (s *SDK) TracerProvider() oteltrace.TracerProvider

TracerProvider returns the SDK's tracer provider interface. Use this to wire the SDK's provider as the global OTel tracer provider if needed, e.g. otel.SetTracerProvider(sdk.TracerProvider()).