observability

README

observability

A self-hosted observability stack for Kubernetes estates, as reusable mechanism: the VictoriaMetrics family as the store, every query scoped at the door to the clusters and namespaces the caller's own token allows, the collectors that stamp those under OpenTelemetry's names, and the alerting rules that catch a backup, a store or a volume failing while everything still looks green.

Artifact	What	Status
charts/observability-crds	The CustomResourceDefinitions the rest of the stack needs, owned as their own release rather than as a side effect of whichever chart installed them first: the VictoriaMetrics operator's, and the four Prometheus Operator scrape kinds every component authors its scrape objects in. Applied before the controllers, never pruned.	released
charts/platform-alerts	The rules that fire when something has stopped working silently: a CronJob that is no longer scheduled, a store whose write path has died, a volume that was never mounted, a store approaching its own read-only limit. Every rule carries the incident that earned it and a negative fixture that must fail.	released
charts/observability-stack	One install of the store: VictoriaMetrics, VictoriaLogs and VictoriaTraces behind an authorising proxy that scopes every query to the clusters and namespaces the caller may read; two vmalerts and Alertmanager (one, or a pair) with a deadman that leaves the cluster; karma as the console that silences in a person's name; network policies and backups; optionally Grafana, forwarding the signed-in user's identity. ha makes it half of a zone-redundant pair: a stores-only mode: replica release beside it, reads over both, a vmalert per replica — see docs/high-availability.md.	released
charts/observability-emitters	Per-cluster collection: a metrics agent, a log agent, an OpenTelemetry gateway and an optional kube-state-metrics, node-exporter and blackbox exporter, each stamping the cluster, the namespace and the environment tier under OpenTelemetry's names — from what the collector can see, never from what the application said — and each replicating to every destination (both halves of a pair) with its own on-disk buffer and credential. The gateway's optional external receiver files telemetry from outside the cluster under an identity a route verified — see docs/external-ingest.md.	released
pkg/tenancy (Go)	From a list of principals, render the proxy's user entries or the token claim an issuer mints — one input, both shapes, so the two can never disagree. Refuses a name that could widen a grant rather than escaping it.	released
notifications: in observability-stack	The one router: receiver kinds and how their secret is mounted, the routing shape by cluster × namespace × severity, the message template, and the refusals that retire the blackhole default. The stack's own self-alerts are a separate release.	released
charts/alert-ingress + cmd/alert-ingress	Events born outside the cluster — a threat finding, a root sign-in, a key use, a budget — into the same router: signed notifications from an allow-listed topic, mapped by values, never dropped.	released
cmd/cwlogs-insights-reader	The Logs Insights reader of observability-emitters cloudwatchLogs.readMode: insights: reads an Infrequent Access CloudWatch log group (no FilterLogEvents) and hands the events to the collector beside it. See docs/reference.md.	unreleased
charts/oncall + cmd/telegram-relay	On call: GoAlert (schedules, escalation, acknowledgement, its alert log as the incident record) fed by Alertmanager, and a relay that turns GoAlert's webhook contact method into a Telegram message to the person paged. No database and no secret value in the chart. See docs/oncall.md.	unreleased
pkg/statusbox, pkg/statusbox/ec2 (Go, Pulumi)	The watcher outside every cluster: several Gatus instances on one immutable EC2 box behind a tunnel and a private network — the deadman, outside-in probes, public status pages per company, and the bridge from an internal alert to a status component.	released
charts/observability-dashboards	The generic dashboards, shipped to wherever Grafana runs, and the lint every dashboard passes: a datasource variable, a cluster variable, the cluster in the title.	released
charts/observability-grafana	One Grafana as the read UI over several stores: a datasource set per store, OIDC sign-in, a database of its own, dashboards from git only. See docs/grafana.md.	released
charts/observability-mcp + cmd/mcp-aggregator	Read-only Model Context Protocol connectors so an agent can read the estate: one per Victoria store (metrics, logs and traces tools in one server, prefixed, allowlisted) and one for Grafana's dashboards. Each is the stock server(s) on loopback behind a proxy that validates the caller's access token and holds the only credential the store sees. See docs/mcp.md.	released
cmd/tenancy-authz (opt-in tenancy.authz in charts/observability-proxy)	An Envoy external authorisation service that unions a person's groups into the vm_access claim vmauth applies, because vmauth can select only one principal per token. See docs/tenancy-authz.md.	released
charts/observability-rum	Browser telemetry for an estate's web apps: Grafana Faro receivers on a self-hosted Alloy, one per app and stamping its own identity, writing to the OTLP gateway with an error fingerprint on every exception so "issues" are built on the log store, with alerts for new and regressed issues, two dashboards, and source maps served from an OCI registry by smctl serve. See docs/frontend.md.	released
lambdaext (Go) + cmd/otlp-lambda	The AWS Lambda extension that exports a function's OpenTelemetry data with its IAM role's identity and no stored secret: STS identity token, RFC 8693 exchange at the issuer, a loopback OTLP/HTTP proxy that adds the bearer, and Lambda platform events as OTLP logs. Released as otlp-lambda-layer_<version>_linux_<arch>.zip with checksums.txt. See docs/integrations/aws-lambda.md.	released
charts/observability-portal + apps/portal	The entrypoint page of an estate: a generic built-in catalog plus the entries and sections an estate adds through values, loaded at runtime from /config/portal.json. A TypeScript, React and Vite app behind an unprivileged nginx, published as its own image. See docs/portal.md.	released
charts/observability-alerting	The alerting plane of an observability-stack install as a chart of its own: the vmalerts, the Alertmanager and its notifications, karma, the Watchdog rule and the store self-alerts, under the same object names the stack renders, so an Application switch adopts them in place. The stack renders them by default; alerting.source: chart hands them over. See docs/adoption.md (0.71.0).	released
charts/observability-proxy	The Door plane of an observability-stack install as a chart of its own: the vmauth proxy and its VMUsers, tenancy-authz with its SecurityPolicy, and the proxy's NetworkPolicies, under the object names the stack's Door always had, so an Application switch adopts them in place. It is the only chart that renders them: observability-stack renders no Door object, and tenancy.identities is the one list of who may read what. See docs/adoption.md (0.81.0) and docs/proxy.md.	unreleased

Charts publish to oci://ghcr.io/truvity/charts/<chart> on every tag; from the release that adds it, the same tag is the Go module github.com/truvity/observability's version. cmd/alert-ingress and cmd/mcp-aggregator publish alongside them, to ghcr.io/truvity/observability/alert-ingress and .../mcp-aggregator at the same tag — charts/alert-ingress's image.tag defaults to it, so an install that sets nothing pulls the matching build.

Who it is for

A platform team running the VictoriaMetrics family on Kubernetes with the VictoriaMetrics operator, an OIDC issuer of their own, and more than one team looking at the same telemetry. It assumes kube-state-metrics and kubelet metrics exist — charts/observability-emitters' kubeStateMetrics can be that source, off by default so a cluster that already has a shared one is never handed a second — and that vmalert selects rules and Alertmanager routes them.

The stores themselves are not here: these charts wrap the upstream ones rather than replacing them. What is here is everything an install is judged on when it fails — whether a query can reach another team's data, whether the write path stopping is noticed, and whether a backup that silently stopped two days ago is still reported healthy.

The model

Two nouns. The scoping key is the cluster and the namespace: every series, log stream and span carries k8s.cluster.name and k8s.namespace.name (spelled k8s_cluster_name and k8s_namespace_name where a label cannot hold a dot, and kubernetes.pod_namespace on logs, where the container-log agent's own spelling is the one both log writers use), stamped by the collectors from the pod they resolved the data to and never from what the application said. A grant names a cluster and the namespaces on it; a project or a team is a derivation from a name to a namespace list that lives with whoever writes grants, and is not a label anywhere. An install is one set of stores serving many of them; isolation happens at query time, where a proxy reads the caller's token and injects the filters that token is entitled to.

The environment tier, deployment.environment.name, rides on every signal and is never a key: two clusters can share a tier, so a filter on it would hand a principal both.

That first sentence is the whole security property, and charts/observability-emitters is what holds it up: the metrics agent discards a k8s_namespace_name label a target exported itself, the gateway strips the one an SDK set before resolving the pod, and the values that would turn either of those off are refusals rather than defaults. Query-time isolation over telemetry an application labelled itself is not isolation.

Everything else follows from those two. Redundancy is the writer's job because no store here replicates across a zone: two independent releases of the stores in two zones, every collector sending to both with a buffer and a credential per destination, the proxy reading its own store first and the peer on failure. High availability is a values flag (ha.enabled, mode: replica, remote.replicas), not a different architecture.

And one router. Every alert an install evaluates and every event the cloud publishes about it reach a person through the same Alertmanager, routed by cluster × namespace × severity; the only things outside that router are the ones that must notice the router itself has died — a watcher on a box outside every cluster, and the edge provider watching the box. docs/target-state.md draws it: every emitter, both halves of a pair, the readers, and the three parties that watch each other.

Install and a worked example

The stack itself installs after the CRDs and before the rules — see docs/adoption.md, which lists what must already exist (the CRDs chart, cert-manager, an OIDC issuer, and the Secret holding the stores' own credentials):

helm install observability oci://ghcr.io/truvity/charts/observability-stack \
  --version <version> --namespace observability --values values.yaml

The rules are their own release:

helm install platform-alerts oci://ghcr.io/truvity/charts/platform-alerts \
  --version <version> --namespace observability --values values.yaml

# values.yaml — a worked example with neutral values.
#
# There is no default store list and the chart refuses to render without
# one: a guessed metric name renders cleanly and then never fires, which
# is the failure this chart exists to prevent. Read these names off your
# own stores' /metrics.
stores:
  - name: metrics
    rowsMetric: vm_rows_inserted_total
    freeSpaceMetric: vm_free_disk_space_bytes
    freeSpaceLimitMetric: vm_free_disk_space_limit_bytes
  - name: logs
    rowsMetric: vl_rows_ingested_total

# vmalert selects rule objects by these.
ruleLabels:
  vmalert: platform

# Whatever the Alertmanager routing tree reads.
commonLabels:
  k8s_cluster_name: example-cluster
  deployment_environment_name: development

runbookBaseUrl: https://runbooks.example.com

Scoping a query to its caller

cfg := tenancy.Config{
    ClaimName:      "groups",
    Audience:       "example-observability-client",
    MetricsBackend: "http://metrics.example:8428",
    LogsBackend:    "http://logs.example:9428",

    // The keys default to what charts/observability-emitters stamps:
    // k8s_cluster_name / k8s_namespace_name on metrics, k8s.cluster.name /
    // kubernetes.pod_namespace on logs. Set them only for collectors that
    // were not built here.

    Principals: []tenancy.Principal{
        {Group: "example:k8s:viewer", Grants: []tenancy.Grant{
            {Cluster: "example-cluster", AllNamespaces: true},
        }},
        {Group: "example:example-app:deployer", Grants: []tenancy.Grant{
            {Cluster: "example-cluster", Namespaces: []string{"example-app"}},
        }},
    },
}

vmauth, err := cfg.RenderVMAuth("https://issuer.example")   // the proxy's config
claim, err := cfg.RenderClaim(cfg.Principals[1])            // what an issuer mints

Both come from the same input on purpose. A difference between them is a difference between what a token says a person may read and what the proxy lets them read, and that is not a difference anyone notices until it matters.

Audience is the client id this proxy's own tokens are minted under, and it is required. vmauth validates a token's expiry and, under OIDC discovery, its issuer, and nothing else — it has no audience option and never inspects aud — so without the pin any unexpired token that issuer minted is admitted whatever client it was minted for, including one the same person holds for a different application. It is rendered into every user's match_claims beside the group, under aud, and both values are escaped and anchored there — vmauth compiles a match_claims value as a regular expression, and neither a client id nor a group name is ours to choose.

Each route RenderVMAuth emits carries the filter argument that applies the claim — .../?extra_filters={{.MetricsExtraFilters}} for metrics, extra_stream_filters={{.LogsExtraStreamFilters}} for logs — because that substitution is the only thing vmauth does with a vm_access claim. A route without it forwards every query unfiltered while the claim beside it still states the grant, which is why a read route cannot be constructed without one. TracesBackend needs AllowUnfilteredTraceReads with it: the trace store's select APIs accept no argument to put a filter in, so that route cannot be scoped and has to be admitted by name. docs/safety.md has the mechanism and the test that catches it.

Consumers

• truvity/gitops: charts observability-crds, observability-stack, observability-emitters

Neighbours

access-roster is the issuer for vmauth tenancy tokens that scope queries to a caller's grants. OpenBao is a relying party: it holds the collector bearer, the webhook secrets and the store credentials this repository's charts take as Secret names, delivered through External Secrets.

Documentation

• docs/target-state.md — what this repository is when complete, the boundary between it and an estate, and the whole of what a consumer writes.
• docs/adoption.md — prerequisites, install order, the zero-diff gate, proving a rule before trusting it, and every breaking upgrade with its steps.
• docs/safety.md — every refusal and every default, with the failure that earned it.
• docs/tenancy-owner.md — the opt-in owner label that routes an alert to the owning company, and what it does not cover.
• docs/high-availability.md — the pair: a stores-only mode: replica release beside a primary with ha.enabled, reads over both, a vmalert per replica, and the runbook for replacing a replica.
• docs/reference.md — every value: default, type, what it does, when it is required.
• docs/doctrine.md — what this repository owns, what the consuming estate owns, and why the shape is what it is.
• docs/notifications.md, docs/alert-ingress.md, docs/oncall.md, docs/statusbox.md, docs/dashboards.md, docs/grafana.md, docs/mcp.md, docs/frontend.md, docs/kube-state-metrics.md — one design page per piece: the values it takes, what it renders, what it refuses, how it is proven.
• docs/external-ingest.md — OTLP from outside the cluster (an AWS Lambda): identity from the gateway's headers, never the payload.
• docs/integrations/aws-lambda.md — the Lambda extension layer: telemetry with the function role's identity, its settings, the IAM it needs and how to attach the layer.
• docs/emitting.md — for whoever wires a service's SDK: the one address, which attributes are theirs and which are taken from them, and how to ask the store rather than trust a 200.

The rule that makes this repository public

Mechanism only. Nothing here names an account, a cluster, a hostname, a bucket, an issuer or a secret path. Every such thing is an input with a neutral default, supplied by the consuming estate from its own private repository, and hack/leak-canary.sh enforces it in CI as its own job — because public history cannot be unpublished.

Secrets are the caller's: a chart takes the name of a Secret and never stores one, generates one into a manifest, or knows a secret manager.

Licences, and the Enterprise boundary

This repository is MIT. It wraps the community edition of the VictoriaMetrics family, which is Apache 2.0 and free to use for any number of tenants or companies. It never pulls an Enterprise image, never renders a licence flag, and never uses an Enterprise-only feature — the charts refuse rather than warn. docs/doctrine.md lists the boundary; docs/safety.md lists what is deliberately absent because of it.

Grafana and its VictoriaMetrics data source plugin are AGPL-3.0; they are referenced by name and never vendored here. Grafana Alloy (Apache-2.0) is the exception that proves the rule's reason: charts/observability-rum vendors the unmodified upstream chart as an archive, pinned, with the Apache-2.0 text in LICENSES/ and the dependency recorded in its Chart.yaml.

Status

Used in production by its maintainers: every chart and package in the table above is released and in use, the store pair included.

Development

devbox shell
just check      # lint, golden renders, leak canary, dashboard lint, rule check
just golden     # regenerate the golden renders — review the diff
just crds       # re-fetch observability-crds from its pinned upstreams

Every chart carries a strict values.schema.json, golden renders under tests/golden/, and one negative fixture per refusal under tests/invalid/. A refusal without a fixture is a refusal that will quietly stop working.

Releasing

One tag stamps every artifact: pushing vX.Y.Z releases every chart at X.Y.Z and, once it exists, the Go module at vX.Y.Z. Chart versions are committed as 0.0.0 and stamped by the release workflow.

Auto-release is present but disarmed, and when armed it only ever cuts patches. Minors, majors and every first release are manual tags, pushed after the CHANGELOG heading for that version has merged.

Licence

MIT, see LICENSE.