k8s-lookout

module
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 26, 2026 License: Apache-2.0

README

k8s-lookout

Deterministic, token-dense eyes on Kubernetes/GKE clusters for LLM-driven troubleshooting agents — the data-plane companion to core-agent.

One multicall binary, lookout, with two halves:

  • Read-path — one-shot diagnostic commands an agent runs mid-investigation (bundle, health, triage …, state …, …). Each emits compressed, secret-safe logfmt/JSON findings instead of raw telemetry dumps, and always ends with an explicit summary line so "cluster healthy" is never ambiguous silence.
  • Watch-path — lookout watch, a resident per-cluster sentinel that turns leading indicators (state transitions, trend slopes, expiry countdowns) into per-incident agent sessions with warm context — catching issues before, or as, they happen rather than after.

Not predictive/ML: every leading indicator is deterministic arithmetic. The complete specification is docs/DESIGN.md; for where things live in the tree, see the repo architecture map.

CI License

Install

Container images (multi-arch amd64 + arm64; distroless static; Sigstore-signed):

docker pull ghcr.io/go-steer/lookout:latest       # default: GCP-free, runs on any conformant cluster
docker pull ghcr.io/go-steer/lookout:latest-gke   # same binary + GKE/GCP provider (-tags allproviders)

The image's ENTRYPOINT is lookout watch, drop-in image-swap compatible with ghcr.io/go-steer/k8s-event-watcher deployments. Verify signatures with:

cosign verify ghcr.io/go-steer/lookout:vX.Y.Z \
  --certificate-identity-regexp '^https://github.com/go-steer/k8s-lookout' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

From source (Go 1.26+):

go install github.com/go-steer/k8s-lookout/cmd/lookout@latest

Quickstart — read-path

Every read command works against your current kubeconfig context, no deployment needed. "Any issues with this cluster?" in one call:

lookout health

kind=health.category severity=info reason=Unavailable message="requires cloud provider metrics (M4); no cloud provider configured" category=control-plane status=unavailable
kind=health.category severity=info category=nodes status=healthy
kind=health.category severity=warning category=crashloops status=degraded total=8 top="pod.restarts agent-sandbox-system/agent-sandbox-controller-7c69875fcc-n7xms; pod.restarts kube-system/coredns-7d764666f9-g82j9; …"
kind=health.category severity=info category=pending status=healthy
kind=health.category severity=info category=rollouts status=healthy
…
kind=pod.restarts severity=warning namespace=kube-system kind_of_object=Pod name=coredns-7d764666f9-g82j9 reason=ExcessiveRestarts fingerprint=sha256:e094a6ee… category=crashloops container=coredns restarts=62
scanned=16 findings=18 elapsed=537ms

(Real output against a kind cluster, abridged.) Healthy resources are omitted; categories still answer explicitly. lookout bundle --workload=Deployment/prod/api is the first call of an incident: one correlated snapshot instead of 4–5 separate reads. Exit 0 is pure payload on stdout; diagnostics go to stderr only.

Quickstart — sentinel

The watch-path deploys per cluster from deploy/:

kubectl apply -f deploy/

That gives the sentinel a dedicated ServiceAccount with a minimum-necessary, read-only ClusterRole (it observes; all mutations happen through the core-agent daemon's own permission gate). Point --daemon-url at your core-agent daemon in deploy/51-deployment-watcher.yaml. Namespace-scoped (Role-only) and project-tier (quota source, -gke image) deployments are covered in DESIGN.md §11 — sources whose RBAC scope isn't satisfied fail loudly at startup, never as a silently empty watch.

Command surface

Command What it does
watch resident sentinel: nine signal sources → filter → dedup → storm correlation → severity routing → enrichment → session injects
mcp serve every read command 1:1 as MCP tools (stdio or localhost HTTP) — how a distroless daemon calls lookout
bundle first call of every incident: sanitized spec + abnormal objects + broken edges + blast radius + distilled logs, one payload
health ten-category cluster scorecard, merged with open sentinel findings and triage-status records
triage delta one scan → everything abnormal: broken workloads, aged Pending, node pressure, gridlocked PDBs, degraded add-ons, hit quotas
triage logs template-fingerprint log dedup: ~150k tokens of logs → ~350
triage events deduped chronological event timeline over the owner-reference tree
triage top CPU/mem saturation vs limits, right now
triage radius blast radius of a workload from the topology index; --at answers "at incident onset"
triage changes "what changed before onset": rollouts, config updates, rescales, scoped to the graph neighborhood
triage spec kubectl describe for agents: sanitized, token-dense, --diff against graph history
triage status read/write §9.4 triage-status records so later scans report triaged reality, not a fresh unknown
state edges dependency-graph verification: config/secret keys, selectors, endpoints, TLS expiry
state webhooks admission webhooks failing closed with dead backends
state wi † GKE Workload Identity KSA↔GSA binding verification
state volumes RWO multi-attach / cross-zone PV locks
stab drift out-of-band drift vs the GitOps manager via managedFields
stab drain everything that will block a node drain
perf probe † control-plane metric packs: apiserver, apf, etcd, startup
cloud stockout|orphans|ipspace|quota † GCP-side reads: zonal stockouts, orphaned disks/LBs, CIDR utilization, quota headroom
net probe active DNS/TCP/HTTP checks from inside the cluster

† needs a cloud provider (the -gke image / -tags gke build). ~80% of the suite is pure client-go and runs on any conformant cluster; the default build links zero GCP SDKs. Provider-gated commands never break or lie on vanilla clusters — they emit an explicit finding and exit 0:

kind=cloud.unavailable severity=info reason=CapabilityUnavailable message="cloud quota needs the provider quota capability: no cloud provider configured" capability=quota provider=none
scanned=0 findings=1 elapsed=0s unavailable="no cloud provider configured"

Agent education ships in skills/: k8s-triage, cluster-health, gitops-drift, and per-symptom playbooks/ — skills teach the decision tree across commands and install into the consuming deployment's .agents/skills/.

Ecosystem

Repo Role
core-agent the agent daemon; lookout talks to it over POST /sessions + /inject
ax fleet layer; joins lookout's frozen signal schema (fingerprint + cluster/project/zone) across clusters

Documentation

License

Apache 2.0 — see LICENSE.

Directories

Path Synopsis
cmd
lookout command
Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`.
 Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`.
dev
tools/harvest-corpus command
harvest-corpus is the §9.3 drill tool over pkg/corpus: it reads a captured inject stream (dev/drills/stub-daemon.py's log format — `kubectl logs` of the stub pod verbatim, optionally interleaved with exported §9.4 triage-status record JSON lines) and emits one labeled trajectory per incident session as JSON lines on stdout, complete trajectories first.
 harvest-corpus is the §9.3 drill tool over pkg/corpus: it reads a captured inject stream (dev/drills/stub-daemon.py's log format — `kubectl logs` of the stub pod verbatim, optionally interleaved with exported §9.4 triage-status record JSON lines) and emits one labeled trajectory per incident session as JSON lines on stdout, complete trajectories first.
internal
mcpserver
Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help.
 Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help.
sitedoc
Package sitedoc generates the docs-site Reference section under docs/site/src/content/docs/reference/ — the fourth generated surface after --help, the MCP schemas, and the skill reference stubs (DESIGN.md §4.4.3: one source of truth, generated outward).
 Package sitedoc generates the docs-site Reference section under docs/site/src/content/docs/reference/ — the fourth generated surface after --help, the MCP schemas, and the skill reference stubs (DESIGN.md §4.4.3: one source of truth, generated outward).
sitedoc/gen command
Command gen regenerates the docs-site Reference section under docs/site/src/content/docs/reference/ from the pkg/checks registry, the sentinel flag/metric inventories, and the signal-schema v1 ledger.
 Command gen regenerates the docs-site Reference section under docs/site/src/content/docs/reference/ from the pkg/checks registry, the sentinel flag/metric inventories, and the signal-schema v1 ledger.
skilldoc
Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas (DESIGN.md §4.4.3: one source of truth, generated outward).
 Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas (DESIGN.md §4.4.3: one source of truth, generated outward).
skilldoc/gen command
Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry.
 Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry.
watch
Command k8s-event-watcher is the v2.6 semi-autonomous-triage sidecar.
 Command k8s-event-watcher is the v2.6 semi-autonomous-triage sidecar.
pkg
checks
Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3).
 Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3).
checks/bundle
Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident.
 Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident.
checks/checktest
Package checktest is the §13 contract-test scaffold for read-path commands.
 Package checktest is the §13 contract-test scaffold for read-path commands.
checks/cloudcheck
Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads.
 Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads.
checks/delta
Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else.
 Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else.
checks/events
Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher.
 Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher.
checks/health
Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard.
 Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard.
checks/logs
Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path.
 Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path.
checks/netprobe
Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all.
 Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all.
checks/perf
Package perf implements `lookout perf probe` (DESIGN.md §5): control-plane and startup performance via data-driven metrics query packs — apiserver p99 by verb/resource, APF queue saturation + 429 rejects, etcd fsync p99 + DB size, pod-startup p95 trend.
 Package perf implements `lookout perf probe` (DESIGN.md §5): control-plane and startup performance via data-driven metrics query packs — apiserver p99 by verb/resource, APF queue saturation + 429 rejects, etcd fsync p99 + DB size, pod-startup p95 trend.
checks/stab
Package stab implements the `lookout stab` command group (DESIGN.md §5): stability reads.
 Package stab implements the `lookout stab` command group (DESIGN.md §5): stability reads.
checks/state
Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification.
 Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification.
checks/top
Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW.
 Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW.
checks/triage
Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes).
 Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes).
cloud
Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification) goes through the Provider interface defined here.
 Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification) goes through the Provider interface defined here.
corpus
Package corpus is the §9.3 verified-fix corpus harvester: it extracts LABELED incident trajectories — symptom → diagnosis → action → externally verified outcome — from a captured inject stream, by pure schema walks over the frozen signal-schema v1 payloads (docs/signal-schema-v1.md).
 Package corpus is the §9.3 verified-fix corpus harvester: it extracts LABELED incident trajectories — symptom → diagnosis → action → externally verified outcome — from a captured inject stream, by pure schema walks over the frozen signal-schema v1 payloads (docs/signal-schema-v1.md).
emit
Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage.
 Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage.
engine
Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents.
 Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents.
graph
Package graph is the in-memory topology index of DESIGN.md §6: a directed, typed graph centered on the Pod, connecting the traffic/policy layers above it (Ingress → Service/EndpointSlice → NetworkPolicy → Pod) to the infrastructure below (Containers, ConfigMaps/Secrets, PVCs, Node, Zone).
 Package graph is the in-memory topology index of DESIGN.md §6: a directed, typed graph centered on the Pod, connecting the traffic/policy layers above it (Ingress → Service/EndpointSlice → NetworkPolicy → Pod) to the infrastructure below (Containers, ConfigMaps/Secrets, PVCs, Node, Zone).
inject
Package inject implements the thin HTTP client the watch sentinel uses to speak to a core-agent daemon, plus the frozen wire types it POSTs (see payload.go).
 Package inject implements the thin HTTP client the watch sentinel uses to speak to a core-agent daemon, plus the frozen wire types it POSTs (see payload.go).
inject/schema
Package schema exports the signal-schema v1 kind ledger as data (docs/signal-schema-v1.md; frozen at M5): every shipped signal kind, the wire struct that serializes it, and its one-line role.
 Package schema exports the signal-schema v1 kind ledger as data (docs/signal-schema-v1.md; frozen at M5): every shipped signal kind, the wire struct that serializes it, and its one-line role.
kube
Package kube provides Kubernetes client bootstrap shared by the lookout subcommands.
 Package kube provides Kubernetes client bootstrap shared by the lookout subcommands.
memory
Package memory holds lookout's durable, agent-queryable memory records (DESIGN.md §9.2/§9.4): the low-volume distilled facts a scheduled distiller pass derives from recurring raw occurrences, and (next change in this stack) the triage-status records incident agents write at material transitions.
 Package memory holds lookout's durable, agent-queryable memory records (DESIGN.md §9.2/§9.4): the low-volume distilled facts a scheduled distiller pass derives from recurring raw occurrences, and (next change in this stack) the triage-status records incident agents write at material transitions.
memory/distill
Package distill is the §9.2 distiller: the scheduled pass in the sentinel that converts recurring raw occurrences (pkg/store) into durable distilled facts (pkg/memory) — "us-east1-b nodegroup n2d-pool: 3 stockouts in 7d".
 Package distill is the §9.2 distiller: the scheduled pass in the sentinel that converts recurring raw occurrences (pkg/store) into durable distilled facts (pkg/memory) — "us-east1-b nodegroup n2d-pool: 3 stockouts in 7d".
sources
Package sources defines the signal-source contract of the sentinel (DESIGN.md §7.2): pluggable sources feeding one shared pipeline — one resident process per cluster, never N sidecars.
 Package sources defines the signal-source contract of the sentinel (DESIGN.md §7.2): pluggable sources feeding one shared pipeline — one resident process per cluster, never N sidecars.
sources/capacity
Package capacity is the capacity signal source (DESIGN.md §7.2 row 7, §10.1): cluster-autoscaler signals from STRUCTURED sources — never the CA text log.
 Package capacity is the capacity signal source (DESIGN.md §7.2 row 7, §10.1): cluster-autoscaler signals from STRUCTURED sources — never the CA text log.
sources/degradation
Package degradation is the degradation signal source (DESIGN.md §7.2 row 5): leading indicators from TRENDS on EndpointSlice ready ratios and from probe flaps below the `Unhealthy` threshold — "payment-backend capacity 5/5 → 3/5 over 10 min".
 Package degradation is the degradation signal source (DESIGN.md §7.2 row 5): leading indicators from TRENDS on EndpointSlice ready ratios and from probe flaps below the `Unhealthy` threshold — "payment-backend capacity 5/5 → 3/5 over 10 min".
sources/expiry
Package expiry is the expiry signal source (DESIGN.md §7.2 row 6): leading COUNTDOWNS — TLS secret certificates, webhook CA bundles, ServiceAccount token expiries where detectable, and cert-manager Certificate status — "cert expires in 72 h and last renewal failed".
 Package expiry is the expiry signal source (DESIGN.md §7.2 row 6): leading COUNTDOWNS — TLS secret certificates, webhook CA bundles, ServiceAccount token expiries where detectable, and cert-manager Certificate status — "cert expires in 72 h and last renewal failed".
sources/k8sevents
Package k8sevents is the first signal source (DESIGN.md §7.2): the core/v1 Event informer that was internal/watch's watcher — the M0 k8s-event-watcher — refactored behind the pkg/sources.Source interface with semantics unchanged.
 Package k8sevents is the first signal source (DESIGN.md §7.2): the core/v1 Event informer that was internal/watch's watcher — the M0 k8s-event-watcher — refactored behind the pkg/sources.Source interface with semantics unchanged.
sources/objectstate
Package objectstate is the object-state signal source (DESIGN.md §7.2 row 2): leading indicators from STATE TRANSITIONS, observed by shared informers on Pods, Nodes, Deployments, EndpointSlices, and PodDisruptionBudgets.
 Package objectstate is the object-state signal source (DESIGN.md §7.2 row 2): leading indicators from STATE TRANSITIONS, observed by shared informers on Pods, Nodes, Deployments, EndpointSlices, and PodDisruptionBudgets.
sources/quota
Package quota is the quota signal source (DESIGN.md §7.2 row 8, §10.2): a leading countdown over cloud quota exhaustion, deployed ONCE PER GCP PROJECT — fifty clusters in a project must not each poll the quota APIs, so Scope() is Project (§11) and exactly one sentinel per project enables this source.
 Package quota is the quota signal source (DESIGN.md §7.2 row 8, §10.2): a leading countdown over cloud quota exhaustion, deployed ONCE PER GCP PROJECT — fifty clusters in a project must not each poll the quota APIs, so Scope() is Project (§11) and exactly one sentinel per project enables this source.
sources/rollout
Package rollout is the rollout signal source (DESIGN.md §7.2 row 3): as-it-happens leading indicators from Deployments and StatefulSets with in-progress rollouts.
 Package rollout is the rollout signal source (DESIGN.md §7.2 row 3): as-it-happens leading indicators from Deployments and StatefulSets with in-progress rollouts.
sources/saturation
Package saturation is the saturation signal source (DESIGN.md §7.2 row 4): trend leading indicators from continuously sampled resource usage — "pod hits memory limit in ~14 min" (slope → ETA), "PVC full in ~3 h".
 Package saturation is the saturation signal source (DESIGN.md §7.2 row 4): trend leading indicators from continuously sampled resource usage — "pod hits memory limit in ~14 min" (slope → ETA), "PVC full in ~3 h".
sources/tokenburn
Package tokenburn is the token-burn signal source (DESIGN.md §7.2 row 9, §12): token spend as a first-class saturation dimension — a runaway agent loop is an OOM in the currency that matters.
 Package tokenburn is the token-burn signal source (DESIGN.md §7.2 row 9, §12): token spend as a first-class saturation dimension — a runaway agent loop is an OOM in the currency that matters.
store
Package store is the sentinel-local raw-occurrence store (DESIGN.md §9.1): ONE bounded, TTL'd embedded SQLite database, living alongside the --dedup-persist file on the same volume, that holds every signal the watch-path pipeline emitted — including info-severity signals that never inject (§7.7) — together with the routing outcome each one received.
 Package store is the sentinel-local raw-occurrence store (DESIGN.md §9.1): ONE bounded, TTL'd embedded SQLite database, living alongside the --dedup-persist file on the same volume, that holds every signal the watch-path pipeline emitted — including info-severity signals that never inject (§7.7) — together with the routing outcome each one received.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.