README
¶
KubeTective
Kubernetes told you the pod was OOMKilled. KubeTective tells you which commit did it.
An incident investigation engine that collects the facts, builds a timeline, ranks the possible causes, and shows every point of confidence as a line of evidence you can read. No LLM in the verdict path — the same incident produces the same answer every time.
Docs · Changelog · Contributing
What it looks like
$ kubectl investigate deployment/checkout --since=30m
╭──────────────────────────────────────────────────╮
│ INCIDENT: deployment/prod/checkout │
│ Status: OOMKILLED │
│ Severity: HIGH │
│ Confidence: 97% │
╰──────────────────────────────────────────────────╯
ROOT CAUSE
Configuration regression: commit 9f2c1a7d (checkout: bump CACHE_SIZE
5000 -> 50000) preceded the failure
EVIDENCE
✓ commit 9f2c1a7d: checkout: bump CACHE_SIZE 5000 -> 50000 (+30)
✓ commit 6 min before onset (+25)
✓ workload observed changed in window (+10)
✓ mechanism: failure follows the change (+30)
RECOMMENDATION
roll back deployment/prod/checkout to the last known-good revision [MEDIUM]
OOMKilled is the symptom, and kubectl already told you that. The answer is
the commit six minutes earlier, and the four evidence terms that got there.
Try it in 2 minutes — no broken cluster needed
git clone https://github.com/GlediLami/kubetective.git && cd kubetective
make build
bin/kubetective replay scenarios/config-regression/record.jsonl
That is the incident above, replayed from a recorded JSONL file. Every scenario
in scenarios/ works the same way — a real investigation you can
re-run without a cluster.
Install
brew install GlediLami/tap/kubetective # Homebrew
go install github.com/GlediLami/kubetective/cmd/kubetective@latest
As a kubectl plugin — put a binary named kubectl-investigate on your PATH:
make install-plugin
kubectl investigate deployment/checkout --since=30m
Building from source needs Go 1.26+. No other build-time dependencies.
Quick start
kubetective investigate deployment/checkout --since=30m # investigate
kubetective incidents # what have I looked at?
kubetective replay <incident-id> # re-run it
kubetective doctor # is everything wired up?
Add evidence sources as you have them — each is optional and degrades quietly:
kubetective investigate deployment/checkout \
--prometheus-url http://localhost:9090 \
--loki-url http://localhost:3100 \
--git-repo ~/code/manifests
Why deterministic
The verdict comes from a rule-based engine, not a language model. Each score is a sum of weighted evidence terms drawn from a documented six-band scale, and every term is printed. An optional LLM layer can rephrase the verdict in plainer language — it can never change a score, invent a cause, or propose an action.
That makes an investigation a test artifact: it replays byte-identically, so it can gate CI. An LLM chat cannot.
What the benchmark actually shows
Four gates run on every commit. These are the real numbers, not aspirations:
17/20 scenarios passed (3 hard-set scenarios are advisory: they calibrate, they do not gate)
mutation gate: 15/15 causal claims held (verdict moves when its evidence is removed)
noise gate: 20/20 verdicts held under 500 irrelevant observations
calibration: 19 ground-truth points (2 incorrect), accuracy 89%
not adopted: out-of-sample ECE 18.6% does not beat the default 7.3%
One of those twenty came off a live cluster rather than a text editor:
live-oom-config-regression is a real
OOMKill traced to the real commit that caused it, recorded from a real API
server and sanitised for publication.
Three things worth saying plainly, because most benchmarks bury them:
Confidence is not currently calibrated, and the engine says so. Expected
calibration error is |confidence − accuracy|, so on a suite the engine never
fails, the error-minimising answer is 100% every time — a fit against such a
suite learns overconfidence, not calibration. Adoption is refused unless the
suite contains real failures, the fit sits inside its search grid, and it beats
the default out-of-sample. Today it refuses on the third. The number you see is
an evidence-margin score at a hand-set temperature until the suite can support
better.
Passing is not the same as reasoning. Each scenario declares what its verdict depends on; the mutation gate deletes that evidence and requires the verdict to move. An engine that keyed on "which analyzer fired" would pass all 20 scenarios and fail this.
The suite is still mostly synthetic. Nineteen of the twenty records are hand-authored miniatures, 4 to 25 observations, where a production namespace carries thousands. The noise gate closes part of that gap and the first live recording closes a little more, but this remains the project's weakest axis.
The most useful thing you can contribute is a real incident, and
kubetective scenario new <incident-id> does the mechanical work: sanitises
the recording, replays it, sweeps the evidence, and drafts the scenario for
you to correct. See scenarios/README.md.
What it finds
11 analyzers: OOM kills, crash loops, image pull failures, unschedulable pods, node pressure, liveness and readiness probes, PVC binding, service selector mismatches, HPA ceilings, DNS failures, and configuration regressions traced to a Git or GitOps commit.
Evidence comes from Kubernetes, Prometheus, Loki, Git, and GitOps controllers (Flux, ArgoCD). Missing sources become visible gaps, never silent ones.
Runs as a CLI, a kubectl plugin, a REST server, or an MCP server. Every
investigation is recorded so it can be replayed, audited, and used as a
benchmark case.
Documentation
| CLI reference | Every command and flag |
| Configuration | Config file, env vars, per-context profiles |
| API reference · OpenAPI spec | REST and MCP servers |
| Architecture | How the pipeline works, and the safety model |
| Alert integrations | PagerDuty, Grafana, Slack — no API keys |
| Benchmark suite | What the four gates measure, and why |
| Comparison | Versus kubectl, LLM chat, and k8sgpt |
Contributing
Good first issues:
- Add a scenario.
kubetective scenario new <incident-id>sanitises a recording, replays it, and drafts the ground truth and mutations for you to correct. It becomes both a demo and a permanent regression test. - Harden an analyzer. Find a false positive against the suite, fix the
scoring, let
kubetective benchmarkprove it. - Add an output format.
jsonandmarkdownexist;sarifandslackare open. - Wire an evidence source (Datadog, Grafana Cloud) behind the collector interface.
make build test vet fmt # the whole loop
kubetective benchmark # must stay green
Adding an analyzer means implementing analyze.Analyzer, registering it in
internal/cli/root.go, and shipping a scenario that proves it. See
CONTRIBUTING.md and SECURITY.md.
License
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
kubectl-investigate
command
kubectl-investigate is the kubectl plugin entry point.
|
kubectl-investigate is the kubectl plugin entry point. |
|
kubetective
command
kubetective is the full KubeTective CLI: investigate, replay, benchmark, doctor.
|
kubetective is the full KubeTective CLI: investigate, replay, benchmark, doctor. |
|
internal
|
|
|
action
Package action implements Phase 3/4 of the remediation model : deterministic preview actions and human-approved application, with audit records appended to the incident file.
|
Package action implements Phase 3/4 of the remediation model : deterministic preview actions and human-approved application, with audit records appended to the incident file. |
|
alert
Package alert turns PagerDuty / Grafana / Slack webhook payloads into an investigation request (roadmap v1.0 integration surfaces).
|
Package alert turns PagerDuty / Grafana / Slack webhook payloads into an investigation request (roadmap v1.0 integration surfaces). |
|
analyze
Package analyze defines the deterministic analyzer contract.
|
Package analyze defines the deterministic analyzer contract. |
|
analyze/configregression
Package configregression implements the config-regression analyzer: it links a change - a git commit touching the workload's manifests or a GitOps reconcile - to the incident onset and builds the "configuration regression" hypothesis.
|
Package configregression implements the config-regression analyzer: it links a change - a git commit touching the workload's manifests or a GitOps reconcile - to the incident onset and builds the "configuration regression" hypothesis. |
|
analyze/crashloop
Package crashloop implements the CrashLoopBackOff analyzer: it activates on container.waiting with reason CrashLoopBackOff (or repeated non-zero exits) and builds the "application crash loop" hypothesis.
|
Package crashloop implements the CrashLoopBackOff analyzer: it activates on container.waiting with reason CrashLoopBackOff (or repeated non-zero exits) and builds the "application crash loop" hypothesis. |
|
analyze/dns
Package dns implements the DNS-failure analyzer: workloads that crash or hang because they cannot resolve names - most commonly because coreDNS / kube-dns is down or the sandbox cannot be created (v0.7: the "why" for crashloops whose events smell of DNS).
|
Package dns implements the DNS-failure analyzer: workloads that crash or hang because they cannot resolve names - most commonly because coreDNS / kube-dns is down or the sandbox cannot be created (v0.7: the "why" for crashloops whose events smell of DNS). |
|
analyze/hpa
Package hpa implements the HorizontalPodAutoscaler analyzer: it activates on hpa.state observations and flags when the workload is pinned at maxReplicas - the capacity-ceiling context that amplifies per-pod failures.
|
Package hpa implements the HorizontalPodAutoscaler analyzer: it activates on hpa.state observations and flags when the workload is pinned at maxReplicas - the capacity-ceiling context that amplifies per-pod failures. |
|
analyze/imagepull
Package imagepull implements the image-pull-failure analyzer: it activates on container.waiting with reason ErrImagePull / ImagePullBackOff and builds the "image cannot be pulled" hypothesis.
|
Package imagepull implements the image-pull-failure analyzer: it activates on container.waiting with reason ErrImagePull / ImagePullBackOff and builds the "image cannot be pulled" hypothesis. |
|
analyze/nodepressure
Package nodepressure implements the node-pressure analyzer: it activates on node.condition observations (MemoryPressure/DiskPressure/PIDPressure) and builds the "node under pressure" hypothesis - the classic root cause that sits above per-pod symptoms.
|
Package nodepressure implements the node-pressure analyzer: it activates on node.condition observations (MemoryPressure/DiskPressure/PIDPressure) and builds the "node under pressure" hypothesis - the classic root cause that sits above per-pod symptoms. |
|
analyze/oom
Package oom implements the memory-exhaustion analyzer: it activates on container.terminated observations with reason OOMKilled, counts them, checks the configured memory limit, and builds the "memory exhaustion" hypothesis with explainable evidence.
|
Package oom implements the memory-exhaustion analyzer: it activates on container.terminated observations with reason OOMKilled, counts them, checks the configured memory limit, and builds the "memory exhaustion" hypothesis with explainable evidence. |
|
analyze/probe
Package probe implements the probe-failure analyzer: it activates on Unhealthy events (liveness/readiness probe failures) and builds the "probe failing" hypothesis - the most common cause of unnecessary restarts and traffic loss.
|
Package probe implements the probe-failure analyzer: it activates on Unhealthy events (liveness/readiness probe failures) and builds the "probe failing" hypothesis - the most common cause of unnecessary restarts and traffic loss. |
|
analyze/pvc
Package pvc implements the PersistentVolumeClaim analyzer: it activates on pvc.state observations and builds the "volume cannot bind" hypothesis when the claim is Pending/Lost or binding events report failures.
|
Package pvc implements the PersistentVolumeClaim analyzer: it activates on pvc.state observations and builds the "volume cannot bind" hypothesis when the claim is Pending/Lost or binding events report failures. |
|
analyze/scheduling
Package scheduling implements the unschedulable-pod analyzer: it activates on pod.state with phase Pending (or FailedScheduling events) and builds the "pod cannot be scheduled" hypothesis with the scheduler's message as the key evidence.
|
Package scheduling implements the unschedulable-pod analyzer: it activates on pod.state with phase Pending (or FailedScheduling events) and builds the "pod cannot be scheduled" hypothesis with the scheduler's message as the key evidence. |
|
analyze/service
Package service implements the service-endpoints analyzer: it activates on service.state observations and builds the "service has no ready endpoints" hypothesis - the 503 / selector-mismatch root cause.
|
Package service implements the service-endpoints analyzer: it activates on service.state observations and builds the "service has no ready endpoints" hypothesis - the 503 / selector-mismatch root cause. |
|
benchmark
Package benchmark implements the scenario benchmark gate: each scenario (scenarios/<name>/) carries a ground-truth spec (scenario.yaml) and a recorded investigation (record.jsonl).
|
Package benchmark implements the scenario benchmark gate: each scenario (scenarios/<name>/) carries a ground-truth spec (scenario.yaml) and a recorded investigation (record.jsonl). |
|
change
Package change implements the "what changed?" detector: it turns observations into ranked Change entries so an investigation can answer "what happened right before the incident?" with a relevance score per change.
|
Package change implements the "what changed?" detector: it turns observations into ranked Change entries so an investigation can answer "what happened right before the incident?" with a relevance score per change. |
|
cli
Package cli implements the kubetective / kubectl-investigate command line.
|
Package cli implements the kubetective / kubectl-investigate command line. |
|
collect
Package collect defines the collector boundary: raw data enters here and is normalized into Observations.
|
Package collect defines the collector boundary: raw data enters here and is normalized into Observations. |
|
collect/git
Package git implements the Git collector: it reads a local repository and emits git.commit observations for commits that touched manifests matching the investigation's target (workload name).
|
Package git implements the Git collector: it reads a local repository and emits git.commit observations for commits that touched manifests matching the investigation's target (workload name). |
|
collect/gitops
Package gitops implements the GitOps collector: it reads Flux Kustomization/HelmRelease and ArgoCD Application custom resources via the dynamic client and normalizes their sync/reconcile state into gitops.state observations - "what the GitOps controller thinks of the workload" .
|
Package gitops implements the GitOps collector: it reads Flux Kustomization/HelmRelease and ArgoCD Application custom resources via the dynamic client and normalizes their sync/reconcile state into gitops.state observations - "what the GitOps controller thinks of the workload" . |
|
collect/kubernetes
Package kubernetes implements the Kubernetes collector: it reads the cluster through the caller's kubeconfig identity (never escalating) and normalizes state, events, container statuses, node conditions, and (optionally) log tails into Observations.
|
Package kubernetes implements the Kubernetes collector: it reads the cluster through the caller's kubeconfig identity (never escalating) and normalizes state, events, container statuses, node conditions, and (optionally) log tails into Observations. |
|
collect/loki
Package loki implements the Loki log collector (v0.8): it serves the adaptive loop's log-evidence requests from a Grafana Loki instance instead of (or in addition to) direct pod-log access via kubectl, which is often restricted in hardened clusters.
|
Package loki implements the Loki log collector (v0.8): it serves the adaptive loop's log-evidence requests from a Grafana Loki instance instead of (or in addition to) direct pod-log access via kubectl, which is often restricted in hardened clusters. |
|
collect/prometheus
Package prometheus implements the Prometheus collector: it queries the Prometheus HTTP API (query_range) for per-container resource series over the investigation window and normalizes them into compact metric.series observations.
|
Package prometheus implements the Prometheus collector: it queries the Prometheus HTTP API (query_range) for per-container resource series over the investigation window and normalizes them into compact metric.series observations. |
|
config
Package config persists the small set of engine settings that calibration can adopt at runtime: currently the calibrated temperature, stored in ~/.kubetective/config.json so every CLI invocation (and the server/MCP modes) scores at the validated temperature.
|
Package config persists the small set of engine settings that calibration can adopt at runtime: currently the calibrated temperature, stored in ~/.kubetective/config.json so every CLI invocation (and the server/MCP modes) scores at the validated temperature. |
|
diag
Package diag implements `kubetective doctor` (issue #2, v1.0 checklist): a read-only environment preflight.
|
Package diag implements `kubetective doctor` (issue #2, v1.0 checklist): a read-only environment preflight. |
|
engine
Package engine orchestrates the investigation pipeline:
|
Package engine orchestrates the investigation pipeline: |
|
graph
Package graph builds the bounded in-memory evidence graph from normalized observations: typed edges (OWNS, RUNS_ON, CHANGED_BEFORE) that the investigation pipeline and the "what changed" ranking both consume .
|
Package graph builds the bounded in-memory evidence graph from normalized observations: typed edges (OWNS, RUNS_ON, CHANGED_BEFORE) that the investigation pipeline and the "what changed" ranking both consume . |
|
hypothesis
Package hypothesis implements the rule-based hypothesis engine: it dedups and merges analyzer-emitted candidates, reranks them, and applies deterministic status rules.
|
Package hypothesis implements the rule-based hypothesis engine: it dedups and merges analyzer-emitted candidates, reranks them, and applies deterministic status rules. |
|
llm
Package llm implements the optional LLM layer: a provider abstraction with an OpenAI-compatible adapter (OpenAI, Ollama, vLLM, llama.cpp), a redacted structured digest builder, and the constrained explainer.
|
Package llm implements the optional LLM layer: a provider abstraction with an OpenAI-compatible adapter (OpenAI, Ollama, vLLM, llama.cpp), a redacted structured digest builder, and the constrained explainer. |
|
logging
Package logging wires the process-wide structured logger (roadmap v1.0 operational maturity).
|
Package logging wires the process-wide structured logger (roadmap v1.0 operational maturity). |
|
memory
Package memory implements incident memory v1 ("seen this before?", roadmap v0.8): incidents are fingerprinted by the SET of observation kinds (the symptom + change shape), and similarity is ranked by Jaccard overlap between those sets.
|
Package memory implements incident memory v1 ("seen this before?", roadmap v0.8): incidents are fingerprinted by the SET of observation kinds (the symptom + change shape), and similarity is ranked by Jaccard overlap between those sets. |
|
model
Package model defines the core KubeTective data model: normalized observations, evidence, the evidence graph, timeline, hypotheses, and incident records.
|
Package model defines the core KubeTective data model: normalized observations, evidence, the evidence graph, timeline, hypotheses, and incident records. |
|
notify
Package notify delivers opt-in completion webhooks for investigations (roadmap v1.0 integration surfaces).
|
Package notify delivers opt-in completion webhooks for investigations (roadmap v1.0 integration surfaces). |
|
recommend
Package recommend implements the deterministic recommendation rule table: the top hypothesis's category maps to a risk-leveled, evidence-linked action (Phase 2, read-only).
|
Package recommend implements the deterministic recommendation rule table: the top hypothesis's category maps to a risk-leveled, evidence-linked action (Phase 2, read-only). |
|
record
Package record persists investigations as append-only JSONL incident records (one Observation per line) - the replay and benchmark substrate.
|
Package record persists investigations as append-only JSONL incident records (one Observation per line) - the replay and benchmark substrate. |
|
redact
Package redact removes identifying and secret material from a recorded incident so it can be shared — attached to a bug report, contributed as a benchmark scenario, or pasted into a postmortem.
|
Package redact removes identifying and secret material from a recorded incident so it can be shared — attached to a bug report, contributed as a benchmark scenario, or pasted into a postmortem. |
|
score
Package score implements the explainable scoring model:
|
Package score implements the explainable scoring model: |
|
server
Package server exposes KubeTective over HTTP (REST) and MCP (stdio) - v0.6 roadmap: "REST API + server mode, MCP server (thin wrapper)".
|
Package server exposes KubeTective over HTTP (REST) and MCP (stdio) - v0.6 roadmap: "REST API + server mode, MCP server (thin wrapper)". |
|
timeline
Package timeline merges observations into a deduplicated, time-sorted, anchored timeline.
|
Package timeline merges observations into a deduplicated, time-sorted, anchored timeline. |
|
pkg
|
|
|
api
Package api is the stable public KubeTective contract.
|
Package api is the stable public KubeTective contract. |
Click to show internal directories.
Click to hide internal directories.