tailscale2otel

README

tailscale2otel — Tailscale → OpenTelemetry & Prometheus exporter

A single Go binary that turns your Tailscale tailnet into OpenTelemetry metrics, logs and traces over OTLP — or a Prometheus /metrics endpoint, or both at once. Network flow logs, configuration audit logs, device fleet health, key expiry, ACL risk, and tailscaled per-node metrics, exported to Grafana Cloud or any OTEL backend. Headscale is supported too.

📖 Full documentation: m7kni.io/tailscale2otel — Getting started · Installation · Configuration · Metrics catalog

255 metrics + 15 log-event types	across 16 collectors
18 Tailscale API endpoints consumed	polled, streamed, or webhook-driven
123 shipped rules	100 alert + 23 recording, Grafana-managed
1 Grafana dashboard	flagship, 10 tabs, v2 dynamic (Grafana 13+)
OTLP push (gRPC/HTTP)	and/or a Prometheus pull endpoint

Why this exists

Tailscale — the WireGuard-based mesh VPN — exposes a genuinely rich observability surface: network flow logs, configuration audit logs, a detailed device inventory, users, keys, DNS, ACL policy, device posture. But it has no Prometheus endpoint of its own, and it streams logs only to SIEM/storage sinks like Splunk or S3. The existing Tailscale exporters cover a slice of the device API and stop there.

tailscale2otel covers the whole surface and models it properly: semantic-convention-compliant OTEL telemetry, with cardinality control that makes flow logs survivable on a metrics backend.

Things nothing else does

• Network flow logs as both metrics and logs. Low-cardinality aggregate counters (tailscale.network.io / .packets / .flows) for dashboards and alerting, plus full-fidelity per-connection records as OTEL logs for drill-down — with a top-N rollup (busiest 500 pairs, rest folded to __other__), opt-in port dimensions, and IANA service-name attribution so dst.port: 443 becomes https. This is the feature that usually makes flow logs unaffordable, and it is the reason this project exists.
• Configuration audit logs → structured OTEL logs + a curated, security-categorized change counter, so you can alert on high-value tailnet changes without ingesting the whole stream.
• Central tailscaled node-metrics polling. Scrapes each node's native client-metrics endpoint (:5252) from one place instead of deploying a scraper per node — with automatic target discovery from the devices API (tag include/exclude, online-only, address family). Emits both the raw tailscaled_* series and 8 curated tailscale.node.* metrics with folded low-cardinality attributes.
• Full API-surface coverage — not just devices. Users, auth keys / OAuth clients / API tokens (with expiry), tailnet settings, DNS, ACL policy (scored for structural risk: wildcards, unrestricted rules, auto-approvers, SSH wildcards), device posture / MDM integrations, Tailscale Services, webhook endpoints, contacts, log-stream delivery health, and OAuth apps.
• Four ingestion paths into one pipeline — poll the API, receive Tailscale's log stream on a built-in Splunk-HEC-compatible receiver, read Tailscale's flow-log export straight out of an S3-compatible bucket, or take real-time HMAC-verified webhooks. All four feed the same processors.
• Offline GeoIP and ASN enrichment of external peers. Optional, from MaxMind .mmdb files on local disk — no hosted lookup service, no per-address network call on the hot path. Country and continent are bounded and can go on flow metrics; the autonomous system (and, with a City database, locality and coordinates) ride the flow logs, where a breakdown costs nothing. Databases hot-swap on a schedule, with a built-in MaxMind updater if you want one. Tailnet addresses are never geolocated.
• Multi-tailnet / MSP mode — one process observing N tailnets, each with its own credentials, and tailscale.tailnet as a real label on every signal (no target_info join required).
• PII redaction on by default — 13 opt-out categories covering emails, user IDs, hostnames, IPs, node IDs and free-text detail, applied to metric attributes, log bodies and span attributes.
• API drift CI. Tailscale's API "may change or break without notice", so a decode-fuzz lane gates every PR and three scheduled lanes diff the live OpenAPI spec, track the upstream client library, and hit the real API read-only. See API drift CI.

Quick start

Docker

docker run --rm \
  -e TS2OTEL_TAILSCALE__TAILNET=example.com \
  -e TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_ID=<client-id> \
  -e TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_SECRET=<client-secret> \
  -e TS2OTEL_OTLP__GRAFANA_CLOUD__INSTANCE_ID=<stack-id> \
  -e TS2OTEL_OTLP__GRAFANA_CLOUD__TOKEN=<token> \
  ghcr.io/rknightion/tailscale2otel:latest

No config file needed — every setting has a TS2OTEL_* environment variable. Mount a YAML file and pass -config /etc/tailscale2otel/config.yaml if you prefer.

Kubernetes (Helm)

Put the credentials in a Secret first, then point the chart at it. They never reach your shell history, and never pass through Helm at all:

cat > creds.env <<'EOF'
TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_ID=...
TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_SECRET=...
TS2OTEL_OTLP__GRAFANA_CLOUD__INSTANCE_ID=...
TS2OTEL_OTLP__GRAFANA_CLOUD__TOKEN=...
EOF
chmod 600 creds.env

kubectl create secret generic tailscale2otel-creds --from-env-file=creds.env
rm creds.env

helm install tailscale2otel oci://ghcr.io/rknightion/charts/tailscale2otel \
  --set-string config.tailscale.tailnet=example.com \
  --set-string existingSecret=tailscale2otel-creds

Do not pass a credential as an inline --set secret.<KEY> value: it lands in your shell history and is visible in ps to every other user on the machine. scripts/check_doc_commands.py fails CI if any documented command does.

See Installation for docker-compose, prebuilt binaries (Linux/macOS/Windows × amd64/arm64), and the full chart values.

Binary

go build -o tailscale2otel ./cmd/tailscale2otel
cp config.example.yaml config.yaml   # then edit; secrets stay in env vars
./tailscale2otel -config config.yaml

./tailscale2otel -version                      # print version and exit
./tailscale2otel -validate -config config.yaml # lint a config without starting

No backend? Run it locally

Set TS2OTEL_OTLP__PROTOCOL=stdout to print metrics and logs to the console.

Where the telemetry goes

• OTLP push (otlp.protocol: grpc|http) with first-class Grafana Cloud support — set otlp.grafana_cloud.{instance_id,token} and the Basic-auth header is built for you. Full TLS/mTLS knobs. Metrics and logs always; traces are opt-in (tracing.enabled) for the exporter's own self-observability, with exemplars linking API-duration histograms to the originating span.
• Prometheus pull endpoint (prometheus.enabled, off by default) — GET /metrics on its own dedicated listener (default :2112), served alongside OTLP push, with optional bearer/basic auth and TLS. Use it if you already run Prometheus and don't want an OTLP pipeline.
• stdout for local debugging.

OTLP→Prometheus naming: query the normalized name. Dots become underscores, monotonic counters gain _total, units suffix (By→_bytes, s→_seconds), and a unit-1 gauge gains _ratio — so tailscale.network.io → tailscale_network_io_bytes_total. The full mapping is in the metrics catalog.

Collectors

Collector	Cadence	Emits
devices	60s	online/last-seen/key-expiry/update gauges, NAT & connectivity quality, per-DERP latency, subnet routes, tailnet lock, fleet hygiene roll-ups. Feeds the enrichment cache
flowlogs	60s	aggregated traffic counters + per-connection flow logs
auditlogs	60s	audit-event logs + a categorized change counter
users	300s	user/role/status counts, per-user device & connection gauges, outstanding invites
keys	300s	expiry gauges and counts across auth keys, OAuth clients and API tokens
oauth_apps	300s	OAuth-application inventory (alpha API; idles silently where unavailable)
settings	600s	tailnet feature-toggle gauges
acl	600s	ACL size, change detection (by ETag), structural risk scoring
dns	600s	nameserver / search-path / split-zone counts, MagicDNS flag
contacts	600s	contact verification status (the email itself is never emitted)
webhooks	600s	webhook-endpoint inventory + per-endpoint subscription counts
posture_integrations	600s	MDM/EDR integration counts, sync health, matched devices
log_stream	600s	Tailscale's own SIEM-sink delivery health + delivery counters
services	600s	Tailscale Services (VIP) inventory — counts, ports, opt-in backing hosts
node_metrics	60s	(opt-in) scrapes tailscaled /metrics endpoints; see above

Each can be disabled or re-tuned. Under provider: headscale the Tailscale-only collectors auto-disable and a reduced set (devices, users, keys, ACL, node-metrics) runs.

Device enrichment depends on the devices collector — flow/audit IP→name resolution silently degrades to unknown/external without it.

Logs: poll, stream or object store — pick one

Both flowlogs and auditlogs take a source of poll (default), stream, objectstore or both. Tailscale exports each log type to object storage independently, so each has its OWN destination (collectors.flowlogs.objectstore / collectors.auditlogs.objectstore) and nothing is inherited between them. Pick exactly one method per log type — running two risks double-counting, cross-source de-dup is only a best-effort failsafe, and the exporter WARNs at startup when it sees this.

# Poll: tailscale2otel pulls on a schedule (interval/lag/initial_lookback/max_window apply).
flowlogs: { enabled: true, source: poll, interval: 60s, lag: 120s, initial_lookback: 5m, max_window: 1h }

# Stream: Tailscale pushes to the built-in HEC receiver (the window fields are ignored).
flowlogs: { enabled: true, source: stream, log_mode: per_connection }

# Object store: read the export Tailscale writes to S3. No API quota, and the cheapest path for a
# busy tailnet. Credentials come from the ambient chain (env, IRSA, instance profile).
flowlogs:
  enabled: true
  source: objectstore
  objectstore: { endpoint: https://s3.eu-west-2.amazonaws.com, region: eu-west-2, bucket: my-flow-logs }

Object-store delivery is at-least-once. With the file checkpoint store, successful object identities and failed-object gaps survive restart; transient failures retry with bounded backoff, while invalid compressed objects are quarantined for operator acknowledgement. One object is all-or-nothing — every row is decoded before any is committed, so a mid-object failure emits nothing rather than a partial prefix — but the object as a unit replays if the process dies between emission and the checkpoint write. OTLP/backend acknowledgement is outside this boundary.

Backfill has a hard ceiling of 14 day partitions — today plus the previous 13 days — under the default layout: partitioned, whatever initial_lookback says. It is permanent, not per-cycle: one cycle enumerates at most 14 day prefixes newest-first, and the cursor only moves forward, so older days are never listed and are skipped with no gap, no error and no metric. layout: flat has no partitions to cap and reaches arbitrarily far back, at the cost of more LIST requests. The exporter warns at startup when initial_lookback exceeds the ceiling. See Streaming & webhooks for the full path-by-path compatibility, delivery and durability matrix.

Checkpoints persist how far poll and object-store collectors have read. Details on all paths, receiver auth, object-gap handling, and auto_configure are in Streaming & webhooks.

Dashboards, alerts & the admin UI

• Dashboards — deploy/grafana/ ships two dashboards on Grafana's v2 schema (Grafana 13+): Tailnet (is my tailnet healthy — devices, network, security, policy) and Exporter health (is the exporter healthy — collection, ingestion, delivery, runtime, cost), cross-linked to each other, with dynamic rendering so a section only appears when its data is present. Grafana 13+ is a hard requirement — 12.4 accepts the file with a 200 and renders nothing, and 11.5 rejects it with the misleading Dashboard title cannot be empty. Push them with gcx resources push -f. See Dashboards.
• Alerts — deploy/alerts/grafana-managed/ ships 100 alert rules and 23 recording rules (123 total) as rules.alerting.grafana.app manifests, one JSON per rule. Push them with gcx resources push -p deploy/alerts/grafana-managed. Every alert carries a runbook_url, and 77 of 78 link a canonical dashboard panel. See Alerts and Runbooks.
• Admin status page — on by default at :9091. Liveness/readiness probes at /healthz and /readyz (never auth-gated), a live status page at /, and the same snapshot at /api/status.json: per-collector health, active-series cardinality with per-label breakdown, the full metrics/log catalog, discovered node targets, and a redacted config summary. Entirely self-contained — no CDN assets, so it renders on an air-gapped tailnet. Auth fails closed on a non-loopback bind with no admin.auth.token.
• Continuous profiling is opt-in — pprof on the admin server (for Grafana Alloy to pull), or push to Pyroscope / Grafana Cloud Profiles.

Configuration

Layered, lowest precedence first: built-in defaults → optional YAML file → environment variables. Every field is settable as TS2OTEL_ + the dotted key path with __ between levels:

Config key	Environment variable
tailscale.auth.oauth.client_secret	TS2OTEL_TAILSCALE__AUTH__OAUTH__CLIENT_SECRET
otlp.endpoint	TS2OTEL_OTLP__ENDPOINT
collectors.flowlogs.interval	TS2OTEL_COLLECTORS__FLOWLOGS__INTERVAL

An unrecognised TS2OTEL_* variable is logged as a WARN at startup — usually a typo.

Authentication: prefer an OAuth client (auto-refreshing, least-privilege all:read) over an API key. Keyless workload identity (OIDC token exchange, e.g. a Kubernetes projected service-account token) is also supported, and every secret has a *_file variant for Docker/Kubernetes secrets.

→ Full configuration reference · every TS2OTEL_* variable · config.example.yaml

Documentation

Getting started	Zero to first metrics in Grafana Cloud
Installation	Docker, Helm, compose, binaries
Configuration	Every key, default and gotcha
Metrics catalog	All 186 metrics and 13 log events
Node metrics	Central tailscaled scraping
Streaming & webhooks	HEC receiver and webhooks
Architecture	How it fits together
Security	Data handling, PII, receiver auth
Troubleshooting	When it doesn't work

Development

go build ./... && go vet ./... && go test -race ./...
golangci-lint run

Small single-purpose packages under internal/: telemetry (OTEL facade), collector (scheduler/registry/checkpoints + one package per source), tsapi (Tailscale client), provider/hsapi (control-plane abstraction + Headscale), flowlog/audit (records + processors), enrich (device cache), rdns, config, and the stream/webhook receivers. Four committed files are generated — run scripts/regen-generated.sh before committing changes that touch them.

API drift CI

Tailscale's API and OpenAPI spec evolve continuously ("may change or break without notice"), which has broken decoders here before. Eight lanes guard it:

Lane	When	What it checks
Schema-driven decode tests	every PR (gates)	synthesizes payloads from the vendored OpenAPI spec + known wire quirks (numeric proto, polymorphic audit old/new) through the real decoders, plus a boundary matrix running every consumed operation against every boundary shape — null, empty container, nulled nullable fields, extreme values, an unknown enum member, an additive field, and a wrong container shape that must be rejected. Runs inside the normal go test -race ./... leg, which ci-success requires
Exploratory fuzzing	every PR (advisory)	go test -fuzz over the HEC envelope, HEC timestamps and the flow/audit decoders. Deliberately not required: finding a NEW crasher is nondeterministic, so gating it would let an unrelated PR randomly block merges. Each target's seed corpus runs in the gated leg above, so a KNOWN crasher still blocks
OpenAPI drift	daily	diffs the live spec against the vendored copy, scoped to consumed operations. Covers response fields, path/query/header parameters (requiredness, type, default, enum), the success-status set and request/response media types, classifying each as breaking, behavioral or additive
Client-lib tracking	weekly	builds and tests against tailscale-client-go/v2@main and @latest
Scheduled fuzzing	weekly	the same nine fuzz targets for 15 minutes each instead of 120 seconds, where a nondeterministic finding costs nobody a blocked merge. Opens a deduplicated tracking issue on a crasher and attaches the failing input
Live contract	daily	hits the real API read-only and asserts every consumed GET still decodes
Changelog review	monthly	reads Tailscale's changelog feed for entries that name something this exporter collects and carry no recorded verdict in spec/changelog-reviewed.json. Catches a capability announced before, or without, any OpenAPI change. Reviewing an entry means recording a verdict — including a negative one, so a surface already declined is never re-proposed
IANA registry freshness	monthly	regenerates the embedded IANA service-name table from the live registry and reports a diff. The committed copy has no other drift gate and its staleness is invisible at runtime — an unregistered port and a port missing from a stale table both map to no service name
Release completeness	every release	reads the published release back and fails when its asset manifest is short. Two releases shipped permanently incomplete behind green workflows before this existed

Scheduled lanes are advisory — they open a deduplicated tracking issue and fail the run, but never block PRs. Of the PR-time lanes, only the schema-driven decode tests gate; exploratory fuzzing does not, for the reason in its row.

The cadences and the advisory-versus-gating split in this table are asserted by internal/ci/workflowcontract_test.go, which reads the workflow files — two of these rows claimed "weekly" against daily crons until that test was added (#436).

Maintainer one-time setup

gh label create api-drift -c FBCA04
gh label create clientlib-drift -c FBCA04
gh label create live-contract -c FBCA04

The live lane stores no long-lived Tailscale key. It runs on a standard GitHub-hosted runner and mints a short-lived token from Tailscale's OAuth endpoint using a read-only (all:read) OAuth client, whose TS_OAUTH_CLIENT_ID / TS_OAUTH_CLIENT_SECRET are repo secrets. Keeping them as secrets is safe because this lane is schedule + workflow_dispatch only, so a fork PR can never run it and never reach them; the minted token is masked and lives only for that run. Set the repo variable TS_TAILNET (the tailnet name is not a secret). Missing configuration fails the lane loudly rather than self-skipping, so a misconfigured preflight cannot look green. Optionally set the ANTHROPIC_API_KEY secret for Claude enrichment on the spec-drift and live lanes; the client-lib lane never receives it by design, since it builds untrusted upstream code.

License

Apache License 2.0 — full text in LICENSE; third-party attribution and bundled notices/SBOMs in LICENSING.md.