cf_observability

README

caerus-framework-observability

Caerus Framework — observability component.

Exposes:

• Kubernetes health-check endpoints aggregating every registered component that implements the optional caerusframework.HealthProvider interface. Components that do not implement it are simply not included, so supporting health checks is entirely optional.
• A /metrics endpoint (Prometheus text format) with Go runtime metrics plus the live state of every registered component implementing the optional cf.MetricsProvider interface.
• OpenTelemetry tracing: when an OTLP endpoint is configured, the component builds a tracer provider, installs it as the global provider (components trace via otel.Tracer), and flushes it at Shutdown.

Endpoints

Endpoint	Probe	Behaviour
/healthz, /livez	liveness	200 ok while the process is alive; component health is deliberately excluded (a dependency outage should make a pod unready, not restartable).
/readyz	readiness (and startup)	200 ok when every registered HealthProvider component is healthy, 503 otherwise, with one fail: <component>: <reason> line per failing check.
/metrics	scrape	Prometheus text format: Go runtime + process metrics and one <name> sample per MetricsProvider component that has something to report.

Kubernetes only inspects the status code: 2xx = healthy, anything else = unhealthy. Configure the probes in the pod spec, e.g.:

livenessProbe:
  httpGet: { path: /healthz, port: 9090 }
  initialDelaySeconds: 3
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /readyz, port: 9090 }
  initialDelaySeconds: 3
  periodSeconds: 5

Point a Prometheus scraper at /metrics, and a collector (e.g. otel-collector) at the configured trace_endpoint for OTLP/gRPC spans.

Wiring

Observability is always-on core, not a chassis component you list next to postgres. cf.New(&cf.FrameworkOptions{Observability: …}) registers it. RunWithSignals (the serve path) starts its Runnable, which is when the process binds the operator HTTP port.

Golden path (app main)

fw := cf.New(&cf.FrameworkOptions{
	Logs: &cf.LogsSettings{
		Format: "json", Level: "info", ConfigSource: "logs",
	},
	Observability: &cf.ObservabilitySettings{
		Bind:         ":9090", // /livez, /readyz, /metrics — bound in Run
		ConfigSource: "observability",
	},
	Components: []cf.CaerusComponent{
		// postgres, valkey, app, …
	},
})
if err := fw.RunWithSignals(ctx, cf.WithShutdownTimeout(15*time.Second)); err != nil {
	log.Fatal(err)
}

The seed’s ConfigSource is the configuration source name (file path flag --observability when it is "observability"). The component Name() is also "observability". Prefer matching those two strings so GetDependencies and the --<name> flag are the same word.

Simple path (tests / one-off binary)

fw := caerusframework.New() // no FrameworkOptions: add core by hand
fw.AddComponent(cf_logs.New(cf_logs.WithWriter(os.Stdout)))
fw.AddComponent(cf_observability.New()) // health + metrics, bind :9090 in Run
// ... register the rest of the components ...
fw.Run(ctx)

Serving vs jobs

This component is the process’s operator shop window (/livez, /readyz, /metrics). It is not the public API server (caerus-framework-http is).

Init prepares collectors, the mux, and the tracer. Run is the only place that calls net.Listen. Framework jobs (--postgresql.job=migrate, fw.RunJob) never start Runnables, and observability is not always initialized on a job (core job Init is logs + configuration). A migrate Job therefore does not open :9090.

Wrong: Init opens :9090 because “health must exist before Run.”
Right: Init prepares; Run binds. Jobs skip Runnables, so one-shot work has
       no operator HTTP.

Configure Kubernetes probes and Prometheus scrapes on serving pods, not on Job pods. /metrics has no scrape auth in this module.

Who may hit :9090 (NetworkPolicy)

Default bind is :9090 (all interfaces). The shop window is then reachable from anything that can route to the pod IP on that port — kubelet probes, an in-cluster Prometheus, and any other pod unless you restrict it. That is an ops-plane problem, not something this module solves with mTLS.

Path A — Cluster scrape (recommended for serve Deployments):

Keep bind: ":9090" so kubelet and Prometheus can reach the pod IP. Allow ingress TCP 9090 only from:

• the nodes / kubelet (liveness and readiness probes), and
• the namespace (or PodSelector) that runs Prometheus / Grafana Alloy.

Deny 9090 from the public Ingress and from app namespaces that have no reason to scrape. Example shape (adjust labels to your cluster):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: observability-shop-window
spec:
  podSelector:
    matchLabels:
      app: myapp          # serving pods only, not migrate Jobs
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: monitoring   # Prometheus / Alloy
        # kubelet probes often come from the node; some CNIs need:
        # - ipBlock: { cidr: <node-pod-CIDR or hostNetwork> }
      ports:
        - protocol: TCP
          port: 9090

Path B — Loopback only (laptop / go run):

{ "bind": "127.0.0.1:9090" }

Nothing on the cluster network can scrape or probe that port. Do not use Path B on a Kubernetes serve pod if you still want /readyz and /metrics from kubelet/Prometheus.

Wrong: all-interfaces bind and no NetworkPolicy, then treating /metrics as “internal” because the Service is ClusterIP.
Right: Path A bind + Policy, or Path B only when the process is local.

Optional component health checks

A component opts in by implementing cf.HealthProvider; the observability component discovers it and folds its health into /readyz:

// Health implements cf.HealthProvider. nil = healthy.
func (c *CFMongoDB) Health(ctx context.Context) error {
    return c.pool.Ping(ctx) // or any other liveness/readiness signal
}

Every check is bounded by the health-check timeout (default 2s); a component that misses its deadline is reported as timed_out so a hung check can never hang the probe.

Optional component metrics (lazy pickup)

A component opts in by implementing cf.MetricsProvider; observability registers a collector for it and reads Metrics() on every /metrics scrape, so the values are always live:

// Metrics implements cf.MetricsProvider.
func (l *Logs) Metrics() []cf.Metric {
    return []cf.Metric{{
        Name:  "logs_info", // served as logs_info
        Value: 1,
        Labels: map[string]string{"format": l.format.String(), "level": l.Level().String()},
    }}
}

Return nil while the component is not initialized (or has nothing to report): the collector then skips it, and the sample appears on the scrape after the component initializes — a lazy pickup with no subscription. Components that do not implement the interface contribute nothing.

Tracing

With an endpoint configured, Init creates an OTLP/gRPC tracer provider and installs it as the global provider; components trace through otel.Tracer. Shutdown flushes pending spans. OTLP uses TLS by default. Set trace_insecure: true only when the collector has no TLS (you are admitting cleartext).

Sampling: head sampling in this process. Default trace_sample_ratio is 1.0 (every new trace is kept; same volume as the old AlwaysSample). Set 0.1 to keep about 10% of new traces. Children follow the parent (ParentBased): if the incoming trace was sampled, this process still records the child. 0 means new traces are not sampled. This is not TLS (trace_insecure) and not /metrics.

Configuration

ObservabilityConfig is file/env-drivable: load it through the configuration component and pass it via WithConfig:

observability:
  health_checks: true          # enable the health-check endpoints
  metrics: true                # enable the /metrics endpoint
  tracing: true                # enable OTLP trace export (needs trace_endpoint)
  bind: ":9090"                # string = one listener; array = multibind
  health_check_timeout_sec: 2  # per-component health check deadline
  trace_endpoint: "otel-collector:4317"
  trace_insecure: false        # default TLS; set true to admit cleartext OTLP
  trace_sample_ratio: 1.0      # 1 = all new traces; 0.1 ≈ 10%; children follow parent
  service_name: myapp          # service.name on exported spans

Option	Default	Purpose
WithHealthChecks(bool)	true	Enable the Kubernetes health-check endpoints.
WithMetrics(bool)	true	Enable the /metrics endpoint.
WithTracing(bool)	false	Enable trace export (active once an endpoint is set).
WithBind(...string)	":9090"	Listen address(es); one string or several host:port.
WithHealthCheckTimeout(d)	2s	Deadline for each component health check.
WithTraceEndpoint(string)	"" (tracing latent)	OTLP/gRPC collector endpoint.
WithTraceInsecure(bool)	false (TLS)	Admit cleartext OTLP.
WithTraceSampleRatio(float64)	1.0	Head sampling 0–1 (ParentBased + ratio).
WithServiceName(string)	"caerus"	service.name attribute on exported spans.
WithConfig(ObservabilityConfig)	—	Loaded config; non-zero fields override the options.
WithConfigSource(string)	""	Bind a configuration source; Init applies its current value and OnConfigReload applies later changes live (tracing) or logs restart-required (bind/metrics/health toggles).
WithLogger(*slog.Logger)	framework logs logger (re-delivered on logs Reconfigure), falling back to slog.Default()	Explicit logger override.

health_checks, metrics and tracing are *bool in ObservabilityConfig so an explicit false in the file is honored (turning the feature off) instead of being treated as "unset".

Component contract

Implements caerusframework.CaerusComponent and cf.Runnable:

• Name() → "observability" (cf_observability.ComponentName)
• GetInitOrderStage() → caerusframework.ObservabilityStage (third bootstrap stage, after logs and configuration)
• GetDependencies() → [logs] (plus configuration when WithConfigSource is set)
• Init sets up the Prometheus registry (Go/process collectors + one collector per registered MetricsProvider), builds and installs the tracer provider when tracing is enabled and an endpoint is configured, and builds the HTTP mux when health checks or metrics are enabled. It does not bind a listen address. With everything disabled it is a no-op besides logger subscribe.
• Run binds the HTTP server (fail-fast on an unusable address) and serves until the framework cancels the run context. Jobs never call Run.
• Shutdown stops the server if one is running, waiting for in-flight requests up to ctx, and flushes the tracer provider.
• Address() returns the bound address (empty when disabled, before Run, or after shutdown) for building probe configs at runtime.
• TracerProvider() returns the configured provider (nil when tracing is inactive).

Docs

• ARCHITECTURE.md — component model and stage ordering.
• LIFECYCLE.md — lifecycle guarantees.

License

Apache License 2.0 — see LICENSE.

Documentation

Index

• Constants
• type Bind
  • func (b Bind) MarshalJSON() ([]byte, error)
  • func (b *Bind) UnmarshalJSON(data []byte) error
  • func (b *Bind) UnmarshalYAML(value *yaml.Node) error
• type Metric
• type MetricType
• type MetricsProvider
• type Observability
  • func New(opts ...Option) *Observability
  • func (c *Observability) Address() string
  • func (c *Observability) CoreConfigSource() ([]cf.ConfigSourceValue, error)
  • func (c *Observability) GetDependencies() []string
  • func (c *Observability) GetInitOrderStage() cf.Stage
  • func (c *Observability) Init(ctx context.Context, fw *cf.CaerusFramework) error
  • func (c *Observability) Name() string
  • func (c *Observability) OnConfigReload(source string, cfg any)
  • func (c *Observability) Run(ctx context.Context) error
  • func (c *Observability) Shutdown(ctx context.Context) error
  • func (c *Observability) TracerProvider() oteltrace.TracerProvider
• type ObservabilityConfig
• type Option
  • func WithBind(addrs ...string) Option
  • func WithConfig(cfg ObservabilityConfig) Option
  • func WithConfigSource(name string) Option
  • func WithHealthCheckTimeout(d time.Duration) Option
  • func WithHealthChecks(enabled bool) Option
  • func WithLogger(logger *slog.Logger) Option
  • func WithMetrics(enabled bool) Option
  • func WithServiceName(name string) Option
  • func WithTraceCAFile(path string) Option
  • func WithTraceEndpoint(endpoint string) Option
  • func WithTraceInsecure(insecure bool) Option
  • func WithTraceSampleRatio(ratio float64) Option
  • func WithTracing(enabled bool) Option

Constants

const ComponentName = "observability"

ComponentName is the framework component name for the observability component. It is the identifier other components use in GetDependencies to require observability.

Types

type Bind

type Bind []string

Bind is one or more host:port listen addresses. JSON/YAML is a string for a single listener (":9090") or an array for several (ports may differ).

func (Bind) MarshalJSON

func (b Bind) MarshalJSON() ([]byte, error)

MarshalJSON writes a string when there is one address, otherwise an array.

func (*Bind) UnmarshalJSON

func (b *Bind) UnmarshalJSON(data []byte) error

UnmarshalJSON accepts a string or an array of strings.

func (*Bind) UnmarshalYAML

func (b *Bind) UnmarshalYAML(value *yaml.Node) error

UnmarshalYAML accepts a scalar or a sequence of host:port strings.

type Metric

type Metric struct {
	// Name is the metric name served on /metrics. It must not be empty for
	// the sample to be emitted.
	Name string
	// Help describes the metric for /metrics consumers.
	Help string
	// Value is the sample value. State/info samples typically use 1 and carry
	// their meaning in Labels.
	Value float64
	// Labels annotate the sample, e.g. {"format": "json", "level": "info"}.
	Labels map[string]string
	// Type selects how the sample is scraped. The zero value (MetricTypeGauge)
	// preserves the pre-existing behavior; set MetricTypeCounter for
	// monotonically increasing event counts.
	Type MetricType
}

Metric is one runtime-state sample a component exposes for the /metrics endpoint. The observability component serves the sample under Name as-is (Name "logs_info" appears in /metrics as "logs_info").

type MetricType

type MetricType int

MetricType distinguishes how a Metric sample is scraped. The zero value (MetricTypeGauge) preserves backward compatibility with existing samples.

const (
	// MetricTypeGauge is the default. The sample is emitted as a Prometheus
	// gauge (current value, goes up and down).
	MetricTypeGauge MetricType = iota
	// MetricTypeCounter marks a monotonically increasing counter. The sample
	// is emitted as a Prometheus counter (only resets on process restart).
	// Components must ensure counter values only increase for the process
	// lifetime.
	MetricTypeCounter
)

type MetricsProvider

type MetricsProvider interface {
	// Metrics returns the component's current runtime state. Return nil while
	// the component is not initialized or has nothing to report.
	Metrics() []Metric
}

MetricsProvider is an optional interface for components that expose runtime state as metrics. The observability component discovers components implementing it and calls Metrics on every /metrics scrape, so the values are always live: a component that is not initialized yet returns nil and is skipped until it does — a lazy pickup that needs no subscription. A component that does not implement MetricsProvider contributes nothing to /metrics.

Bootstrap components (logs, configuration) do not implement this interface to avoid import cycles; observability scrapes their state directly via cf.Get + exported state helpers.

type Observability

type Observability struct {
	// contains filtered or unexported fields
}

Observability is the caerus-framework-observability component. It exposes:

• Kubernetes health-check endpoints (liveness /healthz + /livez, readiness /readyz) that aggregate the health of every registered component implementing cf.HealthProvider. Components that do not implement it are simply not included, so supporting health checks is entirely optional.
• a /metrics endpoint (Prometheus text format) with Go runtime metrics plus the live state of every registered component implementing cf.MetricsProvider. State is read on every scrape, so a component that is not initialized yet returns nil and is skipped until it does (lazy pickup); components that do not implement the interface contribute nothing.
• OpenTelemetry tracing: when an OTLP endpoint is configured, the component builds a tracer provider, installs it as the global provider (components trace via otel.Tracer), and flushes it at Shutdown.

func New

func New(opts ...Option) *Observability

New creates an observability component. Health checks and metrics are on by default; tracing is off until explicitly enabled with a config value or option. Init prepares collectors and the HTTP mux; Run (cf.Runnable) binds the listener when at least one of health checks or metrics is enabled. Jobs never start Runnables, so a migrate/seed process does not open the operator HTTP port.

func (*Observability) Address

func (c *Observability) Address() string

Address returns the bound address of the HTTP server, or "" if neither health checks nor metrics are enabled, Init has not run, or Run has not bound yet. It is useful for building Kubernetes probe configs at runtime after the serve path has started.

func (*Observability) CoreConfigSource

func (c *Observability) CoreConfigSource() ([]cf.ConfigSourceValue, error)

CoreConfigSource implements cf.CoreConfigSource. Observability imports the configuration module (Lookup at Init). Logs cannot (configuration imports logs); this package does. The framework still collects the declaration during argv absorption so ParseFlags sees --observability and OBSERVABILITY_ before Init.

func (*Observability) GetDependencies

func (c *Observability) GetDependencies() []string

GetDependencies implements cf.Dependencies. The component logs through the framework logs component, and depends on configuration when WithConfigSource is set (it reads its own source through the configuration component).

func (*Observability) GetInitOrderStage

func (c *Observability) GetInitOrderStage() cf.Stage

GetInitOrderStage implements cf.CaerusComponent. Observability is part of the bootstrap prefix and initializes after logs and configuration.

func (*Observability) Init

func (c *Observability) Init(ctx context.Context, fw *cf.CaerusFramework) error

Init implements cf.CaerusComponent. It sets up metrics (Prometheus registry with Go runtime collectors and one collector per registered component implementing cf.MetricsProvider) and tracing (OTLP tracer provider, when an endpoint is configured and tracing is enabled). When health checks or metrics are enabled it builds the HTTP mux and discovers registered cf.HealthProvider components. It does not bind a listen address — that is Run's job, so a framework job (which never starts Runnables) does not expose /metrics or probes.

func (*Observability) Name

func (c *Observability) Name() string

Name implements cf.CaerusComponent.

func (*Observability) OnConfigReload

func (c *Observability) OnConfigReload(source string, cfg any)

OnConfigReload implements cf.ConfigReloader. It applies a freshly loaded ObservabilityConfig from the source named by WithConfigSource. Tracing toggle/endpoint changes take effect immediately (provider swap); HTTP endpoint changes (bind address, health-check/metrics toggles) are logged as restart-required and the last-good server keeps running. The initial value delivered by configuration before this component's Init is ignored (Init reads the source itself and must not build providers early).

func (*Observability) Run

func (c *Observability) Run(ctx context.Context) error

Run implements cf.Runnable. It binds the operator HTTP server (health probes and /metrics) and serves until ctx is canceled. Init only prepares the mux; claiming a listen address here means a job-only process — which never starts Runnables — does not expose those endpoints.

func (*Observability) Shutdown

func (c *Observability) Shutdown(ctx context.Context) error

Shutdown implements cf.CaerusComponent. It stops the HTTP server if Run (or an in-flight serve) left one running, waiting for in-flight requests up to ctx, and flushes the tracer provider. Safe to call even if Init never ran or the features are disabled.

func (*Observability) TracerProvider

func (c *Observability) TracerProvider() oteltrace.TracerProvider

TracerProvider returns the OpenTelemetry tracer provider configured at Init, or nil when tracing is disabled or no endpoint was configured. Components trace through otel.Tracer (the provider is installed globally); the accessor is for embedding and for building composite providers.

type ObservabilityConfig

type ObservabilityConfig struct {
	// HealthChecks enables the Kubernetes health-check HTTP endpoints. It is a
	// *bool so an absent key is distinguishable from an explicit
	// health_checks: false, which is required to turn the endpoints off (the
	// component's default is enabled).
	HealthChecks *bool `json:"health_checks,omitempty" yaml:"health_checks,omitempty" env:"HEALTH_CHECKS" flag:"observability-health-checks"`
	// Metrics enables the /metrics endpoint (Prometheus text format; default
	// enabled). It is a *bool so an explicit metrics: false is honored.
	Metrics *bool `json:"metrics,omitempty" yaml:"metrics,omitempty" env:"METRICS" flag:"observability-metrics"`
	// Tracing enables OpenTelemetry trace export over OTLP/gRPC (default off;
	// internal tracing mechanics stay dark until enabled). It is a *bool so an
	// explicit tracing: false is honored.
	Tracing *bool `json:"tracing,omitempty" yaml:"tracing,omitempty" env:"TRACING" flag:"observability-tracing"`
	// Bind is the operator HTTP listen address(es). A JSON string is one
	// listener (":9090"); an array is multibind (ports may differ). Omit keeps
	// the default ":9090".
	Bind Bind `json:"bind,omitempty" yaml:"bind,omitempty" env:"BIND" flag:"observability-bind"`
	// HealthCheckTimeoutSec bounds each component health check (default 2).
	HealthCheckTimeoutSec int `` /* 160-byte string literal not displayed */
	// TraceEndpoint is the OTLP/gRPC collector endpoint (e.g.
	// "otel-collector:4317"). When set and tracing is enabled, spans are
	// exported to it.
	TraceEndpoint string `json:"trace_endpoint,omitempty" yaml:"trace_endpoint,omitempty" env:"TRACE_ENDPOINT" flag:"observability-trace-endpoint"`
	// TraceInsecure admits cleartext OTLP. Default false (TLS). Ops must set
	// true to talk to a collector without TLS.
	TraceInsecure bool `json:"trace_insecure,omitempty" yaml:"trace_insecure,omitempty" env:"TRACE_INSECURE" flag:"observability-trace-insecure"`
	// TraceCAFile is an optional PEM CA file for TLS OTLP (empty = system roots).
	TraceCAFile string `json:"trace_ca_file,omitempty" yaml:"trace_ca_file,omitempty" env:"TRACE_CA_FILE" flag:"observability-trace-ca-file"`
	// TraceSampleRatio is head sampling in this process (0–1). Nil keeps the
	// default 1.0 (same as AlwaysSample). Explicit 0 means new traces are not
	// sampled; children still follow a sampled parent (ParentBased).
	TraceSampleRatio *float64 `` /* 136-byte string literal not displayed */
	// ServiceName is the OpenTelemetry service.name attribute attached to
	// exported spans (default "caerus").
	ServiceName string `json:"service_name,omitempty" yaml:"service_name,omitempty" env:"SERVICE_NAME" flag:"observability-service-name"`
}

ObservabilityConfig is the file/env-drivable observability configuration. Load it through the configuration component and pass it via WithConfig; both JSON and YAML tags are provided.

type Option

type Option func(*options)

Option configures the observability component at construction time.

func WithBind

func WithBind(addrs ...string) Option

WithBind sets one or more host:port listen addresses (default ":9090").

func WithConfig

func WithConfig(cfg ObservabilityConfig) Option

WithConfig sets the configuration loaded from the configuration component. Non-zero fields of cfg override the values set by the convenience options, which act as in-code defaults:

cfg, _ := cf_configuration.Lookup[cf_observability.ObservabilityConfig](conf, "observability")
o := cf_observability.New(cf_observability.WithConfig(*cfg))

func WithConfigSource

func WithConfigSource(name string) Option

WithConfigSource names the configuration source (caerus-framework- configuration) whose ObservabilityConfig is applied to the component at Init and again on every validated reload via OnConfigReload. Prefer this over a WithConfig snapshot when the component is framework-managed: the source stays the live options plane. The component self-registers the source during argv absorption (default file config/<name>.json, owner cf_observability); an argv --<name> file-path override wins, and the app may also register its own Source[ObservabilityConfig] for a custom default. Until the source loads, construction-time defaults apply.

func WithHealthCheckTimeout

func WithHealthCheckTimeout(d time.Duration) Option

WithHealthCheckTimeout sets the deadline for each component health check (default 2s). A component that misses its deadline is reported as not ready.

func WithHealthChecks

func WithHealthChecks(enabled bool) Option

WithHealthChecks enables (default) or disables the Kubernetes health-check HTTP endpoints.

func WithLogger

func WithLogger(logger *slog.Logger) Option

WithLogger overrides the logger used for component diagnostics. By default the component logs through the framework logs component (declared in GetDependencies); WithLogger is an explicit override for tests and embedded use and wins over the framework logger. slog.Default() remains the fallback only when neither is available.

func WithMetrics

func WithMetrics(enabled bool) Option

WithMetrics enables (default) or disables the /metrics endpoint.

func WithServiceName

func WithServiceName(name string) Option

WithServiceName sets the OpenTelemetry service.name attribute (default "caerus").

func WithTraceCAFile

func WithTraceCAFile(path string) Option

WithTraceCAFile sets an optional PEM CA path for TLS OTLP.

func WithTraceEndpoint

func WithTraceEndpoint(endpoint string) Option

WithTraceEndpoint sets the OTLP/gRPC collector endpoint. Only used when tracing is enabled. TLS is the default; set WithTraceInsecure(true) to admit cleartext.

func WithTraceInsecure

func WithTraceInsecure(insecure bool) Option

WithTraceInsecure admits cleartext OTLP when true. Default false (TLS).

func WithTraceSampleRatio

func WithTraceSampleRatio(ratio float64) Option

WithTraceSampleRatio sets head sampling for new traces (0–1, default 1.0). Invalid values fail when the tracer provider is built (Init / tracing reload).

func WithTracing

func WithTracing(enabled bool) Option

WithTracing enables (default off) OpenTelemetry trace export. Tracing is off by default and only active once enabled and an endpoint is set with WithTraceEndpoint (or the loaded config's trace_endpoint).