README
¶
k8s-lookout
Data-plane intelligence for core-agent:
deterministic, token-dense eyes on Kubernetes/GKE clusters for LLM-driven
troubleshooting agents.
Two halves, one multicall binary (lookout):
- Read-path — one-shot diagnostic commands an agent runs mid-investigation
(
lookout triage|state|stab|perf|cloud|bundle|health), emitting compressed, secret-safe, logfmt/JSON findings instead of raw telemetry dumps. - Watch-path —
lookout watch, a resident per-cluster sentinel that turns leading indicators (state transitions, trend slopes, expiry countdowns) into per-incident agent sessions with warm context — detecting issues before, or as, they happen rather than after.
Status: M3 complete. Leading indicators and history have shipped on
top of the M2 closed loop: four new signal sources (rollout —
evidence-based stall detection ahead of progressDeadlineSeconds;
saturation — linear usage-vs-limit forecasts with wire-attached
forecast{eta, confidence_basis}; degradation — ready-ratio trends and
probe-flap counting; expiry — certificate/token countdowns), the §9.1
raw occurrence store (--store: one bounded SQLite recording every signal
with its routing outcome) with §6.6 graph history (compressed snapshots +
a per-delta change log), and the history-consuming read path: triage events|top|radius|changes and net probe, with --at=<instant> --store=<file> answering point-in-time questions offline. Measured in the
exit drills: a staged bad deploy opened a session 3m10s in while every
user request returned 200; a staged memory leak got a critical session 14
minutes before the OOM kill (forecast ETA accurate to 31 s); "blast radius
at onset" was answered 28m34s after the fact from a copied store — see
docs/milestones/M3.md for the evidence
(M2: closed loop;
M1: read-path core;
M0: lookout watch, the moved
k8s-event-watcher, image-swap compatible). Images are published at
ghcr.io/go-steer/lookout. The complete specification is
docs/DESIGN.md; next up is capacity & quota (M4,
§14): the capacity + quota sources, cloud stockout|orphans|ipspace|quota, distilled memories (§9.2), and the
store/memory-merged health.
Roughly 80% of the suite is pure client-go and runs on any conformant
Kubernetes cluster; GKE/GCP-specific capability lives behind a cloud-provider
boundary (DESIGN.md §2).
Ecosystem
| Repo | Role |
|---|---|
core-agent |
the agent daemon; lookout talks to it over POST /sessions + /inject |
ax |
fleet layer; consumes lookout's rollup-ready signal schema across clusters |
License
Apache 2.0 — see LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
lookout
command
Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`.
|
Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`. |
|
internal
|
|
|
mcpserver
Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help.
|
Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help. |
|
skilldoc
Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas (DESIGN.md §4.4.3: one source of truth, generated outward).
|
Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas (DESIGN.md §4.4.3: one source of truth, generated outward). |
|
skilldoc/gen
command
Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry.
|
Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry. |
|
watch
Command k8s-event-watcher is the v2.6 semi-autonomous-triage sidecar.
|
Command k8s-event-watcher is the v2.6 semi-autonomous-triage sidecar. |
|
pkg
|
|
|
checks
Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3).
|
Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3). |
|
checks/bundle
Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident.
|
Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident. |
|
checks/checktest
Package checktest is the §13 contract-test scaffold for read-path commands.
|
Package checktest is the §13 contract-test scaffold for read-path commands. |
|
checks/cloudcheck
Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads.
|
Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads. |
|
checks/delta
Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else.
|
Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else. |
|
checks/events
Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher.
|
Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher. |
|
checks/health
Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard.
|
Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard. |
|
checks/logs
Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path.
|
Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path. |
|
checks/netprobe
Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all.
|
Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all. |
|
checks/state
Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification.
|
Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification. |
|
checks/top
Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW.
|
Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW. |
|
checks/triage
Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes).
|
Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes). |
|
cloud
Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification) goes through the Provider interface defined here.
|
Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification) goes through the Provider interface defined here. |
|
emit
Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage.
|
Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage. |
|
engine
Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents.
|
Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents. |
|
graph
Package graph is the in-memory topology index of DESIGN.md §6: a directed, typed graph centered on the Pod, connecting the traffic/policy layers above it (Ingress → Service/EndpointSlice → NetworkPolicy → Pod) to the infrastructure below (Containers, ConfigMaps/Secrets, PVCs, Node, Zone).
|
Package graph is the in-memory topology index of DESIGN.md §6: a directed, typed graph centered on the Pod, connecting the traffic/policy layers above it (Ingress → Service/EndpointSlice → NetworkPolicy → Pod) to the infrastructure below (Containers, ConfigMaps/Secrets, PVCs, Node, Zone). |
|
inject
Package inject implements the thin HTTP client the watch sentinel uses to speak to a core-agent daemon, plus the frozen wire types it POSTs (see payload.go).
|
Package inject implements the thin HTTP client the watch sentinel uses to speak to a core-agent daemon, plus the frozen wire types it POSTs (see payload.go). |
|
kube
Package kube provides Kubernetes client bootstrap shared by the lookout subcommands.
|
Package kube provides Kubernetes client bootstrap shared by the lookout subcommands. |
|
sources
Package sources defines the signal-source contract of the sentinel (DESIGN.md §7.2): pluggable sources feeding one shared pipeline — one resident process per cluster, never N sidecars.
|
Package sources defines the signal-source contract of the sentinel (DESIGN.md §7.2): pluggable sources feeding one shared pipeline — one resident process per cluster, never N sidecars. |
|
sources/capacity
Package capacity is the capacity signal source (DESIGN.md §7.2 row 7, §10.1): cluster-autoscaler signals from STRUCTURED sources — never the CA text log.
|
Package capacity is the capacity signal source (DESIGN.md §7.2 row 7, §10.1): cluster-autoscaler signals from STRUCTURED sources — never the CA text log. |
|
sources/degradation
Package degradation is the degradation signal source (DESIGN.md §7.2 row 5): leading indicators from TRENDS on EndpointSlice ready ratios and from probe flaps below the `Unhealthy` threshold — "payment-backend capacity 5/5 → 3/5 over 10 min".
|
Package degradation is the degradation signal source (DESIGN.md §7.2 row 5): leading indicators from TRENDS on EndpointSlice ready ratios and from probe flaps below the `Unhealthy` threshold — "payment-backend capacity 5/5 → 3/5 over 10 min". |
|
sources/expiry
Package expiry is the expiry signal source (DESIGN.md §7.2 row 6): leading COUNTDOWNS — TLS secret certificates, webhook CA bundles, ServiceAccount token expiries where detectable, and cert-manager Certificate status — "cert expires in 72 h and last renewal failed".
|
Package expiry is the expiry signal source (DESIGN.md §7.2 row 6): leading COUNTDOWNS — TLS secret certificates, webhook CA bundles, ServiceAccount token expiries where detectable, and cert-manager Certificate status — "cert expires in 72 h and last renewal failed". |
|
sources/k8sevents
Package k8sevents is the first signal source (DESIGN.md §7.2): the core/v1 Event informer that was internal/watch's watcher — the M0 k8s-event-watcher — refactored behind the pkg/sources.Source interface with semantics unchanged.
|
Package k8sevents is the first signal source (DESIGN.md §7.2): the core/v1 Event informer that was internal/watch's watcher — the M0 k8s-event-watcher — refactored behind the pkg/sources.Source interface with semantics unchanged. |
|
sources/objectstate
Package objectstate is the object-state signal source (DESIGN.md §7.2 row 2): leading indicators from STATE TRANSITIONS, observed by shared informers on Pods, Nodes, Deployments, EndpointSlices, and PodDisruptionBudgets.
|
Package objectstate is the object-state signal source (DESIGN.md §7.2 row 2): leading indicators from STATE TRANSITIONS, observed by shared informers on Pods, Nodes, Deployments, EndpointSlices, and PodDisruptionBudgets. |
|
sources/rollout
Package rollout is the rollout signal source (DESIGN.md §7.2 row 3): as-it-happens leading indicators from Deployments and StatefulSets with in-progress rollouts.
|
Package rollout is the rollout signal source (DESIGN.md §7.2 row 3): as-it-happens leading indicators from Deployments and StatefulSets with in-progress rollouts. |
|
sources/saturation
Package saturation is the saturation signal source (DESIGN.md §7.2 row 4): trend leading indicators from continuously sampled resource usage — "pod hits memory limit in ~14 min" (slope → ETA), "PVC full in ~3 h".
|
Package saturation is the saturation signal source (DESIGN.md §7.2 row 4): trend leading indicators from continuously sampled resource usage — "pod hits memory limit in ~14 min" (slope → ETA), "PVC full in ~3 h". |
|
store
Package store is the sentinel-local raw-occurrence store (DESIGN.md §9.1): ONE bounded, TTL'd embedded SQLite database, living alongside the --dedup-persist file on the same volume, that holds every signal the watch-path pipeline emitted — including info-severity signals that never inject (§7.7) — together with the routing outcome each one received.
|
Package store is the sentinel-local raw-occurrence store (DESIGN.md §9.1): ONE bounded, TTL'd embedded SQLite database, living alongside the --dedup-persist file on the same volume, that holds every signal the watch-path pipeline emitted — including info-severity signals that never inject (§7.7) — together with the routing outcome each one received. |
Click to show internal directories.
Click to hide internal directories.