k8s-lookout

module

v0.26.0 Latest Latest Go to latest Published: Sep 21, 2026 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/go-steer/k8s-lookout

Links

Open Source Insights

README ¶

k8s-lookout

Deterministic, token-dense eyes on Kubernetes/GKE clusters for LLM-driven troubleshooting agents — the data-plane companion to core-agent.

One multicall binary, lookout, with two halves:

Read-path — one-shot diagnostic commands an agent runs mid-investigation (bundle, health, triage …, state …, …). Each emits compressed, secret-safe logfmt/JSON findings instead of raw telemetry dumps, and always ends with an explicit summary line so "cluster healthy" is never ambiguous silence.
Watch-path — lookout watch, a resident per-cluster sentinel that turns leading indicators (state transitions, trend slopes, expiry countdowns) into per-incident agent sessions with warm context — catching issues before, or as, they happen rather than after.

Not predictive/ML: every leading indicator is deterministic arithmetic. The complete specification is docs/DESIGN.md; for where things live in the tree, see the repo architecture map.

Documentation: go-steer.github.io/k8s-lookout. For what each of the three modes looks for, one page per mode: What lookout detects. AI agents installing or operating this: start at the agent guide, or fetch the whole docs as one file — llms.txt / llms-full.txt.

Install

Prebuilt binaries (v0.13.0+; Linux/macOS amd64 + arm64, Windows amd64; both flavors; keyless-signed checksums) on the releases page:

gh release download -R go-steer/k8s-lookout -p 'lookout_*_linux_amd64.tar.gz'
tar -xzf lookout_*_linux_amd64.tar.gz && sudo install lookout /usr/local/bin/

Container images (multi-arch amd64 + arm64; distroless static; Sigstore-signed):

docker pull ghcr.io/go-steer/lookout:latest       # default: GCP-free, runs on any conformant cluster
docker pull ghcr.io/go-steer/lookout:latest-gke   # same binary + GKE/GCP provider (-tags allproviders)

The image's ENTRYPOINT is lookout watch, so a Deployment's bare-flag args: splice in behind watch. Verify signatures with:

cosign verify ghcr.io/go-steer/lookout:vX.Y.Z \
  --certificate-identity-regexp '^https://github.com/go-steer/k8s-lookout' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Each image also carries an SPDX SBOM attestation per platform, signed with the same keyless identity — swap verify for verify-attestation --type spdxjson to read it back.

From source (Go 1.26+):

go install github.com/go-steer/k8s-lookout/cmd/lookout@latest

Quickstart — read-path

Every read command works against your current kubeconfig context, no deployment needed. "Any issues with this cluster?" in one call:

lookout health

kind=health.category severity=info reason=Unavailable message="requires cloud provider metrics (M4); no cloud provider configured" category=control-plane status=unavailable
kind=health.category severity=info category=nodes status=healthy
kind=health.category severity=warning category=crashloops status=degraded total=8 top="pod.restarts agent-sandbox-system/agent-sandbox-controller-7c69875fcc-n7xms; pod.restarts kube-system/coredns-7d764666f9-g82j9; …"
kind=health.category severity=info category=pending status=healthy
kind=health.category severity=info category=rollouts status=healthy
…
kind=pod.restarts severity=warning namespace=kube-system kind_of_object=Pod name=coredns-7d764666f9-g82j9 reason=ExcessiveRestarts fingerprint=sha256:e094a6ee… category=crashloops container=coredns restarts=62
scanned=16 findings=18 elapsed=537ms

(Real output against a kind cluster, abridged.) Healthy resources are omitted; categories still answer explicitly. lookout bundle --workload=Deployment/prod/api is the first call of an incident: one correlated snapshot instead of 4–5 separate reads. Exit 0 is pure payload on stdout; diagnostics go to stderr only.

Quickstart — sentinel

The watch-path deploys per cluster from deploy/:

kubectl apply -f deploy/

That gives the sentinel a dedicated ServiceAccount with a minimum-necessary, read-only ClusterRole (it observes; all mutations happen through the core-agent daemon's own permission gate). Point --daemon-url at your core-agent daemon in deploy/51-deployment-watcher.yaml. Namespace-scoped (Role-only) and project-tier (quota source, -gke image) deployments are covered in DESIGN.md §11 — sources whose RBAC scope isn't satisfied fail loudly at startup, never as a silently empty watch.

Want to see both halves against real failures first? examples/ stands up a kind cluster, the sentinel against a capture stub, demo workloads, and ten inject/verify/revert failure scenarios (examples/e2e runs the lot) — plus the recipe for testing the CLI through agent harnesses via skills or MCP.

Command surface

Command	What it does
`watch`	resident sentinel: nine signal sources → filter → dedup → storm correlation → severity routing → enrichment → session injects
`mcp`	serve every read command 1:1 as MCP tools (stdio or localhost HTTP) — how a distroless daemon calls lookout
`scan`	start here: one call runs every target-free incident check, then drills into the dependency edges of whatever it flagged. No target, no flags
`bundle`	first call of every incident: sanitized spec + abnormal objects + broken edges + blast radius + distilled logs, one payload
`health`	ten-category cluster scorecard, merged with open sentinel findings and triage-status records
`triage list`	what exists: `kubectl get` across every kind at once, one line per object, each leading with the `<Kind>/<namespace>/<name>` target the other reads take
`triage delta`	one scan → everything abnormal: broken workloads, aged Pending, node pressure, gridlocked PDBs, degraded add-ons, hit quotas
`triage logs`	template-fingerprint log dedup: ~150k tokens of logs → ~350
`triage events`	deduped chronological event timeline over the owner-reference tree
`triage top`	CPU/mem saturation vs limits, right now
`triage radius`	blast radius of a workload from the topology index; `--at` answers "at incident onset"
`triage changes`	"what changed before onset": rollouts, config updates, rescales, scoped to the graph neighborhood
`triage spec`	kubectl describe for agents: sanitized, token-dense, `--diff` against graph history
`triage status`	read/write §9.4 triage-status records so later scans report triaged reality, not a fresh unknown
`findings diff`	what changed since the previous scan — new, ongoing, escalated, resolved, suppressed — instead of the whole open list again
`findings ack`	suppress one finding for a window after someone has taken it; it comes back on its own when the window expires
`state edges`	dependency-graph verification: config/secret keys, selectors, endpoints, TLS expiry
`state webhooks`	admission webhooks failing closed with dead backends
`state wi` †	GKE Workload Identity KSA↔GSA binding verification
`state volumes`	RWO multi-attach / cross-zone PV locks
`state storage`	why a PersistentVolumeClaim will never bind: missing StorageClass, no cluster default, static-only class, stranded volumes
`state gateway` ‡	the Gateway API path end to end: GatewayClass → Gateway → listener → HTTPRoute → Service, and every hop that is rejected, unprogrammed, or points at nothing
`stab drift`	out-of-band drift vs the GitOps manager via managedFields
`stab drain`	everything that will block a node drain
`perf probe` †	control-plane metric packs: `apiserver`, `apf`, `etcd`, `startup`
`cloud stockout` †	zonal GCE capacity stockouts, with event-derived reroute candidates — the cloud-side why behind pods stuck Pending
`cloud orphans` †	billing-active leftovers: unattached disks, forwarding rules routing to zero endpoints
`cloud ipspace` †	pod/service/node CIDR utilization per subnet — IP space is incompressible
`cloud quota` †	per-project quota usage vs limit, ranked nearest-to-exhaustion
`net probe`	active DNS/TCP/HTTP checks from inside the cluster
`audit workloads`	healthy workloads with no safety net: no PDB, single replica, no probes, no spread, autoscalers that structurally cannot scale
`audit hardening`	workload security posture: privileged containers, host namespaces, hostPath mounts, used default-SA tokens, namespaces with no PSA
`audit netpol`	NetworkPolicy coverage: namespaces nothing isolates, workloads that fell through their neighbours' selectors
`audit cluster` †	GKE cluster security configuration: Workload Identity, legacy metadata endpoints, an internet-reachable control plane
`audit upgrades` †	upgrade and patch readiness: how far behind the control plane and node pools are, and whether anything closes the gap on its own
`audit exemptions`	the `--exemptions` file itself: which reviewed opt-outs have lapsed, and which are about to

The audit rows answer a different question from everything above them: not "what is broken now" but "what has no safety net while it is still healthy" — a standing claim, --exemptions-auditable, meant for a scheduled sweep rather than an incident.

‡ reads an optional API group. Discovery decides: the check auto-enables where the CRDs are installed, and on a cluster without them emits one crd.unavailable finding and exits 0 rather than reporting a clean bill of health it did not earn.

† needs a cloud provider (the -gke image / -tags gke build). ~80% of the suite is pure client-go and runs on any conformant cluster; the default build links zero GCP SDKs. Provider-gated commands never break or lie on vanilla clusters — they emit an explicit finding and exit 0:

kind=cloud.unavailable severity=info reason=CapabilityUnavailable message="cloud quota needs the provider quota capability: no cloud provider configured" capability=quota provider=none
scanned=0 findings=1 elapsed=0s unavailable="no cloud provider configured"

Every finding these commands emit carries a kind=, and every kind they can emit is declared on the command itself — the whole vocabulary, with its claim and its severities, is one table: Finding kinds. A kind missing from a run means the check looked and found nothing; a check that could not run says so instead.

Agent education ships in skills/: k8s-triage, cluster-health, gitops-drift, and per-symptom playbooks/ — skills teach the decision tree across commands and install into the consuming deployment's .agents/skills/.

Ecosystem

Repo	Role
`core-agent`	the agent daemon; lookout talks to it over `POST /sessions` + `/inject`

Documentation

docs/DESIGN.md — the complete, normative specification
docs/repo-map.md — repo architecture map: layout, data paths, frozen contracts
docs/adding-a-check.md — how to add a read-path command, one real check end to end
docs/signal-schema-v1.md — the frozen fleet-rollup wire contract
docs/milestones/ — per-milestone exit evidence (M0–M5 complete, v0.6.0)
CHANGELOG.md

Contributing

New checks are the most common contribution, and docs/adding-a-check.md walks one end to end — dev/tools/new-check scaffolds the rest. CONTRIBUTING.md covers what review will ask about and the two designs that most often look like omissions. ROADMAP.md is where things are going, at the resolution of "which epic".

Missed detections are the most valuable issue we get: the product rests on silence meaning healthy, so a real cluster state we failed to report is evidence rather than a feature request. Security vulnerabilities go privately, never to an issue.

License

Apache 2.0 — see LICENSE.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
lookout command Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`.	Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`.
dev
tools/harvest-corpus command harvest-corpus is the §9.3 drill tool over pkg/corpus: it reads a captured inject stream (dev/drills/stub-daemon.py's log format — `kubectl logs` of the stub pod verbatim, optionally interleaved with exported §9.4 triage-status record JSON lines) and emits one labeled trajectory per incident session as JSON lines on stdout, complete trajectories first.	harvest-corpus is the §9.3 drill tool over pkg/corpus: it reads a captured inject stream (dev/drills/stub-daemon.py's log format — `kubectl logs` of the stub pod verbatim, optionally interleaved with exported §9.4 triage-status record JSON lines) and emits one labeled trajectory per incident session as JSON lines on stdout, complete trajectories first.
internal
mcpserver Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help.	Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help.
newcheck command Command newcheck scaffolds a read-path check: the command file, its test suite, and a first golden.	Command newcheck scaffolds a read-path check: the command file, its test suite, and a first golden.
sitedoc Package sitedoc generates the docs-site Reference section under docs/site/src/content/docs/reference/ — the fourth generated surface after --help, the MCP schemas, and the skill reference stubs (DESIGN.md §4.4.3: one source of truth, generated outward).	Package sitedoc generates the docs-site Reference section under docs/site/src/content/docs/reference/ — the fourth generated surface after --help, the MCP schemas, and the skill reference stubs (DESIGN.md §4.4.3: one source of truth, generated outward).
sitedoc/gen command Command gen regenerates the docs-site Reference section under docs/site/src/content/docs/reference/ from the pkg/checks registry, the sentinel flag/metric inventories, and the signal-schema v1 ledger.	Command gen regenerates the docs-site Reference section under docs/site/src/content/docs/reference/ from the pkg/checks registry, the sentinel flag/metric inventories, and the signal-schema v1 ledger.
skilldoc Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas, and the site reference pages are the fourth (DESIGN.md §4.4.3: one source of truth, generated outward).	Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas, and the site reference pages are the fourth (DESIGN.md §4.4.3: one source of truth, generated outward).
skilldoc/gen command Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry.	Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry.
telemetry Package telemetry initializes OpenTelemetry for the sentinel.	Package telemetry initializes OpenTelemetry for the sentinel.
version Package version centralizes build-identity reporting for cmd/lookout, ported from core-agent's internal/version (issue #146: a go-installed lookout reported the literal string "dev", with no way to correlate a binary with a release).	Package version centralizes build-identity reporting for cmd/lookout, ported from core-agent's internal/version (issue #146: a go-installed lookout reported the literal string "dev", with no way to correlate a binary with a release).
watch Auto defaults for the sentinel (--sources=auto / --storm=auto): probe-and-enable resolution at startup.	Auto defaults for the sentinel (--sources=auto / --storm=auto): probe-and-enable resolution at startup.
pkg
checks Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3).	Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3).
checks/all Package all blank-imports every read-path check package, so that one import populates the default registry with the complete command set:	Package all blank-imports every read-path check package, so that one import populates the default registry with the complete command set:
checks/audit Package audit implements the `lookout audit` command group (docs/fleet-audit-detectors-design.md, epic #182): deterministic best-practice POSTURE checks, as distinct from the incident groups that report what is currently broken.	Package audit implements the `lookout audit` command group (docs/fleet-audit-detectors-design.md, epic #182): deterministic best-practice POSTURE checks, as distinct from the incident groups that report what is currently broken.
checks/bundle Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident.	Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident.
checks/checktest Package checktest is the §13 contract-test scaffold for read-path commands.	Package checktest is the §13 contract-test scaffold for read-path commands.
checks/cloudcheck Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads.	Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads.
checks/crd Package crd is the read path's seam for detectors over API groups that may or may not be installed: Gateway API, OLM, KEDA, Kyverno.	Package crd is the read path's seam for detectors over API groups that may or may not be installed: Gateway API, OLM, KEDA, Kyverno.
checks/delta Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else.	Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else.
checks/events Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher.	Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher.
checks/findings Package findings implements the `lookout findings` command group (docs/findings-diff-design.md, issue #212): the run-to-run transition surface an unattended agent needs so consecutive scans report what CHANGED rather than re-listing the same forty findings every fifteen minutes.	Package findings implements the `lookout findings` command group (docs/findings-diff-design.md, issue #212): the run-to-run transition surface an unattended agent needs so consecutive scans report what CHANGED rather than re-listing the same forty findings every fifteen minutes.
checks/health Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard.	Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard.
checks/inventory Package inventory implements `lookout triage list` (issue #252): kubectl get, aggregated — every namespaced kind an incident normally involves, in one call, one line per object.	Package inventory implements `lookout triage list` (issue #252): kubectl get, aggregated — every namespaced kind an incident normally involves, in one call, one line per object.
checks/logs Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path.	Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path.
checks/netprobe Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all.	Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all.
checks/perf Package perf implements `lookout perf probe` (DESIGN.md §5): control-plane and startup performance via data-driven metrics query packs — apiserver p99 by verb/resource, APF queue saturation + 429 rejects, etcd fsync p99 + DB size, pod-startup p95 trend.	Package perf implements `lookout perf probe` (DESIGN.md §5): control-plane and startup performance via data-driven metrics query packs — apiserver p99 by verb/resource, APF queue saturation + 429 rejects, etcd fsync p99 + DB size, pod-startup p95 trend.
checks/scan Package scan implements `lookout scan`: the zero-argument entry point.	Package scan implements `lookout scan`: the zero-argument entry point.
checks/stab Package stab implements the `lookout stab` command group (DESIGN.md §5): stability reads.	Package stab implements the `lookout stab` command group (DESIGN.md §5): stability reads.
checks/state Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification.	Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification.
checks/top Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW.	Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW.
checks/triage Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes).	Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes).
cloud Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification, cluster-config posture) goes through the Provider interface defined here.	Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification, cluster-config posture) goes through the Provider interface defined here.
corpus Package corpus is the §9.3 verified-fix corpus harvester: it extracts LABELED incident trajectories — symptom → diagnosis → action → externally verified outcome — from a captured inject stream, by pure schema walks over the frozen signal-schema v1 payloads (docs/signal-schema-v1.md).	Package corpus is the §9.3 verified-fix corpus harvester: it extracts LABELED incident trajectories — symptom → diagnosis → action → externally verified outcome — from a captured inject stream, by pure schema walks over the frozen signal-schema v1 payloads (docs/signal-schema-v1.md).
cronsched Package cronsched resolves a CronJob's spec.schedule into activation times.	Package cronsched resolves a CronJob's spec.schedule into activation times.
emit Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage.	Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage.
engine Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents.	Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents.