README
¶
k8s-lookout
Data-plane intelligence for core-agent:
deterministic, token-dense eyes on Kubernetes/GKE clusters for LLM-driven
troubleshooting agents.
Two halves, one multicall binary (lookout):
- Read-path — one-shot diagnostic commands an agent runs mid-investigation
(
lookout triage|state|stab|perf|cloud|bundle|health), emitting compressed, secret-safe, logfmt/JSON findings instead of raw telemetry dumps. - Watch-path —
lookout watch, a resident per-cluster sentinel that turns leading indicators (state transitions, trend slopes, expiry countdowns) into per-incident agent sessions with warm context — detecting issues before, or as, they happen rather than after.
Status: M4 complete. Capacity & quota have shipped on top of the M3
leading indicators: the capacity source (structured cluster-autoscaler
signals — CA events, the status ConfigMap's target/ready gap, provider
scale decisions — never the text log) and the per-PROJECT quota source
(usage-vs-limit slope forecasts: "exhausted in ~6d at current slope",
never just "at 87%"), correlated so a GCE_QUOTA_EXCEEDED scaleup
failure and the quota forecast that predicted it are ONE incident; every
quota.forecast carries a drafted increase request (slope-derived
suggested limit + human-grade justification) that the agent files through
core-agent's permission gate — lookout only drafts, never mutates. Plus
cloud stockout|orphans|ipspace|quota point-in-time reads, distilled
memories (recurring occurrences → durable facts, §9.2), and triage-status
records (§9.4): an incident agent's diagnosis re-routes followups
(downgraded incidents stop re-paging) and lookout health --store
reports triage_status=triaged + root cause at the agent's severity
instead of a fresh unknown — measured mid-crashloop in the exit drill.
See docs/milestones/M4.md for the evidence
(M3: leading indicators + history;
M2: closed loop;
M1: read-path core;
M0: lookout watch, the moved
k8s-event-watcher, image-swap compatible). Images are published at
ghcr.io/go-steer/lookout. The complete specification is
docs/DESIGN.md; next up is fleet & corpus (M5,
§14): the remaining reads (state wi|webhooks|volumes, stab drift|drain, perf probe), the fingerprint schema finalized with AX,
the token-burn source, and the §9.3 corpus harvester contract
validated end-to-end.
Roughly 80% of the suite is pure client-go and runs on any conformant
Kubernetes cluster; GKE/GCP-specific capability lives behind a cloud-provider
boundary (DESIGN.md §2).
Ecosystem
| Repo | Role |
|---|---|
core-agent |
the agent daemon; lookout talks to it over POST /sessions + /inject |
ax |
fleet layer; consumes lookout's rollup-ready signal schema across clusters |
License
Apache 2.0 — see LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
lookout
command
Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`.
|
Command lookout is the single multicall binary of k8s-lookout (DESIGN.md §4.1): deterministic, token-dense reads of Kubernetes/GKE clusters for agent-driven troubleshooting, plus the resident per-cluster sentinel, `lookout watch`. |
|
dev
|
|
|
drills/write-triage-status
command
write-triage-status is a DRILL FIXTURE, not a product surface: it writes one §9.4 triage-status record into a sentinel store the way an incident agent eventually will through core-agent's shared Memory interface.
|
write-triage-status is a DRILL FIXTURE, not a product surface: it writes one §9.4 triage-status record into a sentinel store the way an incident agent eventually will through core-agent's shared Memory interface. |
|
internal
|
|
|
mcpserver
Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help.
|
Package mcpserver serves the registered read-path checks as MCP tools (DESIGN.md §4.3): every non-hidden checks.Command becomes one tool whose name is the command's MCPName, whose description is the command's §4.4.1 micro-skill metadata, and whose input schema is derived mechanically from the same FlagSpecs that generate --help. |
|
skilldoc
Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas (DESIGN.md §4.4.3: one source of truth, generated outward).
|
Package skilldoc generates the per-command reference stubs under skills/<skill>/references/ from the pkg/checks command metadata — the third generated surface after --help and the MCP schemas (DESIGN.md §4.4.3: one source of truth, generated outward). |
|
skilldoc/gen
command
Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry.
|
Command gen regenerates the skill reference stubs under skills/*/references/ from the pkg/checks registry. |
|
watch
Command k8s-event-watcher is the v2.6 semi-autonomous-triage sidecar.
|
Command k8s-event-watcher is the v2.6 semi-autonomous-triage sidecar. |
|
pkg
|
|
|
checks
Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3).
|
Package checks is the read-path command surface: implementations plus the metadata registry that is the single source of truth for every invocation surface (§4.3, §4.4.3). |
|
checks/bundle
Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident.
|
Package bundle implements `lookout bundle` (DESIGN.md §5): the first tool call of every incident. |
|
checks/checktest
Package checktest is the §13 contract-test scaffold for read-path commands.
|
Package checktest is the §13 contract-test scaffold for read-path commands. |
|
checks/cloudcheck
Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads.
|
Package cloudcheck implements the `lookout cloud` command group (DESIGN.md §5): stockout, orphans, ipspace, and quota — the GCP-side point-in-time reads. |
|
checks/delta
Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else.
|
Package delta implements `lookout triage delta` (DESIGN.md §5): one scan of the cluster's current state that reports every abnormal object and nothing else. |
|
checks/events
Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher.
|
Package events implements `lookout triage events` (DESIGN.md §5): the deduped chronological event timeline over a target's owner-reference tree, absorbing v2's ev-sifter and hpa-loop-catcher. |
|
checks/health
Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard.
|
Package health implements `lookout health` (DESIGN.md §5): the "are there issues with this cluster?" scorecard. |
|
checks/logs
Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path.
|
Package logs implements `lookout triage logs` (DESIGN.md §5): the token-density workhorse of the read path. |
|
checks/netprobe
Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all.
|
Package netprobe implements `lookout net probe` (DESIGN.md §5): active DNS/TCP/HTTP checks for hypothesis CONFIRMATION — "is this Service name resolvable", "does this port accept connections", "what does this endpoint actually return" — bending read-only in letter, not spirit: packets are sent, but nothing in the cluster is mutated and no Kubernetes API is touched at all. |
|
checks/perf
Package perf implements `lookout perf probe` (DESIGN.md §5): control-plane and startup performance via data-driven metrics query packs — apiserver p99 by verb/resource, APF queue saturation + 429 rejects, etcd fsync p99 + DB size, pod-startup p95 trend.
|
Package perf implements `lookout perf probe` (DESIGN.md §5): control-plane and startup performance via data-driven metrics query packs — apiserver p99 by verb/resource, APF queue saturation + 429 rejects, etcd fsync p99 + DB size, pod-startup p95 trend. |
|
checks/stab
Package stab implements the `lookout stab` command group (DESIGN.md §5): stability reads.
|
Package stab implements the `lookout stab` command group (DESIGN.md §5): stability reads. |
|
checks/state
Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification.
|
Package state implements the `lookout state` command group (DESIGN.md §5): dependency and configuration verification. |
|
checks/top
Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW.
|
Package top implements `lookout triage top` (DESIGN.md §5): CPU/memory saturation vs limits, RIGHT NOW. |
|
checks/triage
Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes).
|
Package triage implements the graph-backed commands of the `lookout triage` group that consume §6.6 history: `triage radius` (blast radius, live or point-in-time via --at) and `triage changes` (what changed in the window before onset, from the delta log the sentinel writes). |
|
cloud
Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification) goes through the Provider interface defined here.
|
Package cloud is the provider boundary of DESIGN.md §2: everything cloud-touching in lookout (capacity explanations, quota inventory, orphan sweeps, metrics queries, IP-space utilization, stockout extraction, workload-identity verification) goes through the Provider interface defined here. |
|
emit
Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage.
|
Package emit implements the §4.2 output contract shared by every read-path command: findings on stdout as flat, ordered key=value records (logfmt by default, one JSON object per line with --format=json), a mandatory terminating summary line (`scanned=<n> findings=<n> elapsed=<d>`), diagnostics on stderr only, and exit codes 0 data / 1 runtime / 2 usage. |
|
engine
Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents.
|
Package engine implements the watch-path signal pipeline (DESIGN.md §7): the Signal type carried between stages (§8 schema), the frozen cross-cluster Fingerprint, and the reason/namespace filter and rolling-window dedup cache that decide which observed signals become incidents. |
|
graph
Package graph is the in-memory topology index of DESIGN.md §6: a directed, typed graph centered on the Pod, connecting the traffic/policy layers above it (Ingress → Service/EndpointSlice → NetworkPolicy → Pod) to the infrastructure below (Containers, ConfigMaps/Secrets, PVCs, Node, Zone).
|
Package graph is the in-memory topology index of DESIGN.md §6: a directed, typed graph centered on the Pod, connecting the traffic/policy layers above it (Ingress → Service/EndpointSlice → NetworkPolicy → Pod) to the infrastructure below (Containers, ConfigMaps/Secrets, PVCs, Node, Zone). |
|
inject
Package inject implements the thin HTTP client the watch sentinel uses to speak to a core-agent daemon, plus the frozen wire types it POSTs (see payload.go).
|
Package inject implements the thin HTTP client the watch sentinel uses to speak to a core-agent daemon, plus the frozen wire types it POSTs (see payload.go). |
|
kube
Package kube provides Kubernetes client bootstrap shared by the lookout subcommands.
|
Package kube provides Kubernetes client bootstrap shared by the lookout subcommands. |
|
memory
Package memory holds lookout's durable, agent-queryable memory records (DESIGN.md §9.2/§9.4): the low-volume distilled facts a scheduled distiller pass derives from recurring raw occurrences, and (next change in this stack) the triage-status records incident agents write at material transitions.
|
Package memory holds lookout's durable, agent-queryable memory records (DESIGN.md §9.2/§9.4): the low-volume distilled facts a scheduled distiller pass derives from recurring raw occurrences, and (next change in this stack) the triage-status records incident agents write at material transitions. |
|
memory/distill
Package distill is the §9.2 distiller: the scheduled pass in the sentinel that converts recurring raw occurrences (pkg/store) into durable distilled facts (pkg/memory) — "us-east1-b nodegroup n2d-pool: 3 stockouts in 7d".
|
Package distill is the §9.2 distiller: the scheduled pass in the sentinel that converts recurring raw occurrences (pkg/store) into durable distilled facts (pkg/memory) — "us-east1-b nodegroup n2d-pool: 3 stockouts in 7d". |
|
sources
Package sources defines the signal-source contract of the sentinel (DESIGN.md §7.2): pluggable sources feeding one shared pipeline — one resident process per cluster, never N sidecars.
|
Package sources defines the signal-source contract of the sentinel (DESIGN.md §7.2): pluggable sources feeding one shared pipeline — one resident process per cluster, never N sidecars. |
|
sources/capacity
Package capacity is the capacity signal source (DESIGN.md §7.2 row 7, §10.1): cluster-autoscaler signals from STRUCTURED sources — never the CA text log.
|
Package capacity is the capacity signal source (DESIGN.md §7.2 row 7, §10.1): cluster-autoscaler signals from STRUCTURED sources — never the CA text log. |
|
sources/degradation
Package degradation is the degradation signal source (DESIGN.md §7.2 row 5): leading indicators from TRENDS on EndpointSlice ready ratios and from probe flaps below the `Unhealthy` threshold — "payment-backend capacity 5/5 → 3/5 over 10 min".
|
Package degradation is the degradation signal source (DESIGN.md §7.2 row 5): leading indicators from TRENDS on EndpointSlice ready ratios and from probe flaps below the `Unhealthy` threshold — "payment-backend capacity 5/5 → 3/5 over 10 min". |
|
sources/expiry
Package expiry is the expiry signal source (DESIGN.md §7.2 row 6): leading COUNTDOWNS — TLS secret certificates, webhook CA bundles, ServiceAccount token expiries where detectable, and cert-manager Certificate status — "cert expires in 72 h and last renewal failed".
|
Package expiry is the expiry signal source (DESIGN.md §7.2 row 6): leading COUNTDOWNS — TLS secret certificates, webhook CA bundles, ServiceAccount token expiries where detectable, and cert-manager Certificate status — "cert expires in 72 h and last renewal failed". |
|
sources/k8sevents
Package k8sevents is the first signal source (DESIGN.md §7.2): the core/v1 Event informer that was internal/watch's watcher — the M0 k8s-event-watcher — refactored behind the pkg/sources.Source interface with semantics unchanged.
|
Package k8sevents is the first signal source (DESIGN.md §7.2): the core/v1 Event informer that was internal/watch's watcher — the M0 k8s-event-watcher — refactored behind the pkg/sources.Source interface with semantics unchanged. |
|
sources/objectstate
Package objectstate is the object-state signal source (DESIGN.md §7.2 row 2): leading indicators from STATE TRANSITIONS, observed by shared informers on Pods, Nodes, Deployments, EndpointSlices, and PodDisruptionBudgets.
|
Package objectstate is the object-state signal source (DESIGN.md §7.2 row 2): leading indicators from STATE TRANSITIONS, observed by shared informers on Pods, Nodes, Deployments, EndpointSlices, and PodDisruptionBudgets. |
|
sources/quota
Package quota is the quota signal source (DESIGN.md §7.2 row 8, §10.2): a leading countdown over cloud quota exhaustion, deployed ONCE PER GCP PROJECT — fifty clusters in a project must not each poll the quota APIs, so Scope() is Project (§11) and exactly one sentinel per project enables this source.
|
Package quota is the quota signal source (DESIGN.md §7.2 row 8, §10.2): a leading countdown over cloud quota exhaustion, deployed ONCE PER GCP PROJECT — fifty clusters in a project must not each poll the quota APIs, so Scope() is Project (§11) and exactly one sentinel per project enables this source. |
|
sources/rollout
Package rollout is the rollout signal source (DESIGN.md §7.2 row 3): as-it-happens leading indicators from Deployments and StatefulSets with in-progress rollouts.
|
Package rollout is the rollout signal source (DESIGN.md §7.2 row 3): as-it-happens leading indicators from Deployments and StatefulSets with in-progress rollouts. |
|
sources/saturation
Package saturation is the saturation signal source (DESIGN.md §7.2 row 4): trend leading indicators from continuously sampled resource usage — "pod hits memory limit in ~14 min" (slope → ETA), "PVC full in ~3 h".
|
Package saturation is the saturation signal source (DESIGN.md §7.2 row 4): trend leading indicators from continuously sampled resource usage — "pod hits memory limit in ~14 min" (slope → ETA), "PVC full in ~3 h". |
|
sources/tokenburn
Package tokenburn is the token-burn signal source (DESIGN.md §7.2 row 9, §12): token spend as a first-class saturation dimension — a runaway agent loop is an OOM in the currency that matters.
|
Package tokenburn is the token-burn signal source (DESIGN.md §7.2 row 9, §12): token spend as a first-class saturation dimension — a runaway agent loop is an OOM in the currency that matters. |
|
store
Package store is the sentinel-local raw-occurrence store (DESIGN.md §9.1): ONE bounded, TTL'd embedded SQLite database, living alongside the --dedup-persist file on the same volume, that holds every signal the watch-path pipeline emitted — including info-severity signals that never inject (§7.7) — together with the routing outcome each one received.
|
Package store is the sentinel-local raw-occurrence store (DESIGN.md §9.1): ONE bounded, TTL'd embedded SQLite database, living alongside the --dedup-persist file on the same volume, that holds every signal the watch-path pipeline emitted — including info-severity signals that never inject (§7.7) — together with the routing outcome each one received. |
Click to show internal directories.
Click to hide internal directories.