observability

Documentation

Overview

Package observability installs the global OTEL tracer and meter providers, and serves the liveness/readiness endpoints an orchestrator probes.

It lives in the root module so that all five binaries share one implementation: the service (cmd/hippocampus), the ingestor, and the four event-sourcing broker bridges. The integration modules already depend on the root module for the contract, and the root module already carries the OTEL dependencies, so sharing this costs neither side anything - whereas a second copy of exporter wiring in each integration is precisely the kind of thing that drifts.

Index

• Constants
• Variables
• func CycleBuckets() metric.Float64HistogramOption
• func Init(ctx context.Context, cfg Config) (func(context.Context) error, error)
• func LatencyBuckets() metric.Float64HistogramOption
• func UnaryClientMetricsInterceptor(endpoint string) grpc.UnaryClientInterceptor
• func WithGroup(attrs ...attribute.KeyValue) metric.MeasurementOption
• type Check
  • func GRPCHealthCheck(conn grpc.ClientConnInterface) Check
• type Config
• type HealthConfig
• type HealthServer
  • func NewHealthServer(cfg HealthConfig) *HealthServer
  • func (h *HealthServer) Handler() http.Handler
  • func (h *HealthServer) Shutdown(ctx context.Context) error
  • func (h *HealthServer) Start() error

Constants

const GroupAttribute = "hippocampus.group"

GroupAttribute is the tenancy dimension every component stamps on its metrics, and it is deliberately a RESOURCE attribute set once per process rather than an instrument attribute read off each record.

That is the only shape that is safe here. A bridge derives a memory's group from the message subject by default, so on a wildcard subscription the per-record value is unbounded - it would be a new metric stream per subject, which is exactly the high-cardinality attribute this repo's instrumentation rules forbid. Set once per process it is bounded by how many processes are deployed, which is what makes "show me this tenant's ingest" answerable without an unbounded series count. In the fleet-of-edges model each edge IS the tenant, so the two coincide.

Variables

var (
	// LatencyBucketBoundaries suit a request served in-process: an RPC, a store call, one message
	// handled by a bridge. The bottom of the ladder is deliberately below a millisecond, since that
	// is where this service's RPCs actually sit and a quantile is useless if everything is in the
	// first bucket - which is the whole defect above, one order of magnitude down.
	LatencyBucketBoundaries = []float64{
		0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10,
	}

	// CycleBucketBoundaries suit a periodic pass rather than a request: a sleep cycle, an ingestor
	// pass. They run for milliseconds on a small store and can legitimately run for minutes on a
	// large one, so the ladder reaches ten minutes - a cycle that long is a real condition an
	// operator needs to see the shape of, not an outlier to be lumped into +Inf.
	CycleBucketBoundaries = []float64{
		0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600,
	}
)

The explicit bucket boundaries for this repo's duration histograms, in SECONDS.

They exist because the OTel SDK's default boundaries are

0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000

which are chosen for MILLISECONDS. Every duration histogram here records seconds, so against those defaults the first finite bucket is five seconds - coarser than every observation a healthy deployment ever makes. All of them land in one bucket, and histogram_quantile then interpolates linearly across it, so p95 comes back as 0.95 x 5 = 4.75 seconds whatever the real figure is.

That is not merely imprecise, it is actively wrong in a way that had already shipped: the HippocampusHighLatency alert in both rule files fires on p95 > 1s, so it fired permanently on any instance serving traffic - measured against a service whose real mean RPC duration was 0.44 MILLISECONDS - and no amount of the service getting faster could ever have cleared it. Found by the soak harness in demo/soak.sh, which needed a trustworthy sleep-cycle quantile of its own.

Both sets are ordinary Prometheus-shaped ladders. What matters about them is only that they bracket the range the instrument actually observes, so that a quantile is interpolated across a narrow bucket rather than across the whole plausible range.

Functions

func CycleBuckets

func CycleBuckets() metric.Float64HistogramOption

CycleBuckets returns the histogram option carrying CycleBucketBoundaries.

func Init

func Init(ctx context.Context, cfg Config) (func(context.Context) error, error)

Init installs the global OTEL tracer and meter providers according to the configuration and returns a shutdown function that flushes and stops them. Spans are exported over OTLP/gRPC and sampled with a parent-based trace-ID ratio sampler, so the sampling ratio applies to locally started traces while honouring sampling decisions made by callers. An empty endpoint leaves the exporter's own default in place (the OTEL_EXPORTER_OTLP_* environment variables, falling back to localhost:4317).

func LatencyBuckets

func LatencyBuckets() metric.Float64HistogramOption

LatencyBuckets returns the histogram option carrying LatencyBucketBoundaries, so a call site reads as what it is rather than as a slice reference.

func UnaryClientMetricsInterceptor

func UnaryClientMetricsInterceptor(endpoint string) grpc.UnaryClientInterceptor

UnaryClientMetricsInterceptor records the rate, errors and duration of every RPC a client makes, tagged with which endpoint it went to.

It is the client-side counterpart to the service's own RED metrics, and it is deliberately here rather than in each integration: the ingestor dials two instances and the broker bridges dial one, but "how many calls, how many failed, how long did they take" is the same question in both, and two copies of it would drift.

Three things follow the service's rpcmetrics.go rather than being invented here. The `rpc` attribute is the method's short name, never a path carrying ids. `outcome` is three-valued (ok/client_error/server_error) rather than a success bool, so an alert can fire on the far end failing without also firing when this client sends something invalid. And the recording is NOT deferred around the invoker: a panic in a call must not be counted as a success.

func WithGroup

func WithGroup(attrs ...attribute.KeyValue) metric.MeasurementOption

WithGroup builds the measurement option for a recording, adding the tenancy attribute when one is configured.

The group is stamped on the metrics as an INSTRUMENT attribute as well as a resource attribute, and the reason is practical rather than aesthetic: the OTLP-to-Prometheus translation promotes only service.name/service.version/job/instance onto each series and puts every other resource attribute in `target_info`, so slicing by tenant would otherwise need

hippocampus_ingestor_events_total * on(job) group_left(hippocampus_group) target_info

on every query and every alert rule. Costing that out: the value is fixed for the lifetime of the process, so it multiplies the series count by exactly one - it carries none of the cardinality risk that reading a group off each record would, which is why that is the shape this is allowed to take. See GroupAttribute.

Types

type Check

type Check func(ctx context.Context) error

Check reports whether one dependency is usable. A nil error means ready.

func GRPCHealthCheck

func GRPCHealthCheck(conn grpc.ClientConnInterface) Check

GRPCHealthCheck builds a readiness Check that calls the standard gRPC health service on a Hippocampus connection.

It probes grpc.health.v1.Health rather than any Hippocampus RPC on purpose. That service is exempt from the auth interceptor, so the check works whatever tier (or absence) of token the component holds and cannot itself be the thing that fails; it touches no stored data; and on the service side it is driven by the same readiness probe that watches the database, so it reports "this instance can actually serve" rather than merely "the TCP connection came up".

A NOT_SERVING response is a failure, as is any transport error - including Unimplemented, which cmd/hippocampus registers the health service unconditionally to rule out: getting it means the far end is not a Hippocampus instance at all, and reporting ready while pointed at the wrong port is the one answer a probe must never give.

type Config

type Config struct {
	TracingEnabled         bool
	TracingSamplingRatio   float64
	MetricsEnabled         bool
	MetricsIntervalSeconds int
	OTLPEndpoint           string
	OTLPInsecure           bool

	// ServiceName names the component in the telemetry (semconv service.name): "hippocampus" for
	// the service itself, "hippocampus-ingestor", "hippocampus-nats-bridge", and so on. Empty
	// falls back to "hippocampus", preserving the service's own resource attributes exactly.
	ServiceName string

	ServiceVersion string

	// Group is the tenancy label stamped on every metric this process emits. Empty omits the
	// attribute entirely rather than emitting a blank one, so a deployment that does not partition
	// by group produces the same series it always did. See GroupAttribute.
	Group string
}

Config carries the observability settings, read from viper in each binary's main(). Tracing and metrics are independently optional; when both are disabled no providers are installed and the instrumentation throughout falls back to the global no-op providers.

type HealthConfig

type HealthConfig struct {
	// Port is the TCP port to serve on. Zero disables the listener entirely, matching the service's
	// gateway.port convention.
	Port int

	// BindAddress restricts the interface; empty binds all of them.
	BindAddress string

	// Version is reported in the /healthz body, as the service's own /healthz does.
	Version string

	// Component names the binary in log lines and in the probe bodies.
	Component string

	// Checks are the readiness checks, keyed by the dependency name reported in the /readyz body.
	// An empty map makes /readyz equivalent to /healthz - honest for a component with no
	// dependencies, rather than a readiness endpoint that silently means nothing.
	Checks map[string]Check

	// CheckTimeout bounds one check; CacheTTL is how long a result is reused. Non-positive values
	// select the package defaults.
	CheckTimeout time.Duration
	CacheTTL     time.Duration
}

HealthConfig configures the probe listener.

type HealthServer

type HealthServer struct {
	// contains filtered or unexported fields
}

HealthServer serves /healthz (process liveness) and /readyz (dependency readiness) for the client-side daemons - the ingestor and the broker bridges - which otherwise listen on nothing at all and so give an orchestrator nothing to probe.

The split matters and is the same one the service makes: /healthz answers "is this process alive", so a Hippocampus instance being briefly unreachable must NOT make it fail and get the container kill-looped; /readyz answers "can this process do its job right now", which for a bridge or an ingestor means "can I reach both ends".

func NewHealthServer

func NewHealthServer(cfg HealthConfig) *HealthServer

NewHealthServer builds the probe server. It does not listen until Start is called.

func (*HealthServer) Handler

func (h *HealthServer) Handler() http.Handler

Handler is the probe mux, exported so a test can drive the endpoints without binding a port.

func (*HealthServer) Shutdown

func (h *HealthServer) Shutdown(ctx context.Context) error

Shutdown stops the listener. It is safe to call when Start was a no-op.

func (*HealthServer) Start

func (h *HealthServer) Start() error

Start binds the listener and serves in the background. A zero port disables the server and returns nil, so a caller need not branch. The bind failure is returned rather than logged, because a probe port already in use is a configuration error worth failing startup on: silently running without probes is how a deployment ends up believing it has health checks it does not.