opscart-k8s-watcher

README

OpsCart Watcher

Kubectl shows resources. Lens shows state. OpsCart shows what deserves your attention.

Read-only  ·  No agents  ·  Credential-free core  ·  Deploy in 30 seconds

---

What in this cluster deserves attention right now?

kubectl    →  shows resources
Grafana    →  shows metrics
Lens       →  shows cluster state
OpsCart    →  shows what deserves attention first

---

Watch the 5-minute demo →

---

Why OpsCart?

A healthy dashboard does not always mean a healthy cluster.

Metrics show whether services are meeting their SLOs. OpsCart surfaces operational conditions that can remain hidden or fragmented across dashboards—crash-looping workloads, image pull failures, privileged containers, missing NetworkPolicies, unattached storage, and resource waste.

This is not another alert aggregator. OpsCart preserves operational memory across scans: when an incident was first detected, whether it resolved and later reoccurred, how restart behavior changed, and which workload is currently affected. A replacement pod may be only five minutes old while the workload incident has existed for weeks. OpsCart keeps that workload-level history without presenting the new pod as the identity of the incident.

Instead of requiring operators to correlate several dashboards during triage, OpsCart presents a prioritized view of what deserves attention, the observed evidence behind it, and read-only investigation commands for the next step.

OpsCart is designed for platform engineers managing Kubernetes clusters who want faster operational triage without deploying node agents or modifying application workloads.

---

Quick Start

Helm (Recommended)

helm install opscart-watcher ./helm/opscart-watcher \
  --namespace opscart-system \
  --create-namespace

kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80
open http://localhost:8080

Incident history persists on a PVC by default — see the chart README for storage options and minikube-specific notes. The raw manifest below is a quickstart; the Helm chart is canonical. Official container images support both linux/amd64 and linux/arm64.

kubectl

kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml
kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80
open http://localhost:8080

Developer Build

git clone https://github.com/opscart/opscart-k8s-watcher.git
cd opscart-k8s-watcher
go build -o opscart-dashboard ./cmd/opscart-dashboard
./opscart-dashboard --cluster my-cluster --port 8080

---

First Login

OpsCart requires authentication by default. When auth.existingSecret is empty, Helm creates a release-managed Secret and preserves its generated credentials across pod restarts and chart upgrades.

kubectl get secret -n opscart-system opscart-watcher-auth \
  -o jsonpath='{.data.username}' | base64 --decode
echo
kubectl get secret -n opscart-system opscart-watcher-auth \
  -o jsonpath='{.data.password}' | base64 --decode
echo

To supply credentials yourself, create a Secret with username and password keys and set auth.existingSecret:

kubectl create secret generic opscart-auth \
  --from-literal=username=admin \
  --from-literal=password=<your-password> \
  -n opscart-system

auth:
  existingSecret: "opscart-auth"

Standalone execution without configured credentials still generates and logs a random password at startup; that password changes when the process restarts.

For team deployments, authenticate at the ingress layer instead with oauth2-proxy — supports Azure AD, Google, GitHub, and generic OIDC. See Security for the full pattern and threat model.

---

Features

Overview

The entry point isn't a KPI grid — it's a written assessment. Situation Briefing states what's actually wrong in plain sentences ("fraud-detection has been crash-looping for 7 days and its restart rate is accelerating"), not a wall of gauges. Below it:

• Operational Memory scoreboard — total incidents ever seen, resolved, reopened, currently accelerating, longest-running incident, most unstable namespace — numbers only possible because OpsCart remembers, not just observes
• Top 5 Things To Fix — ranked by severity and restart rate, each with a memory line (first detected, reopen count, trend) and a direct link to investigate
• Cluster Health, Namespace Health, and Security Status at a glance

No competitor in this space — Grafana, Lens, k9s — can produce any of this, because none of them remember anything between scans.

Operational Triage

Incident Score — A single 0–100 score derived from crash loops, image pull failures, security posture, waste, and network policy gaps. Trend arrows and a 7-point sparkline show whether the cluster is getting better or worse.

War Room — Every critical incident in one view, prioritized by severity and restart rate. Detects crash loops, image pull failures, probe failures (containers repeatedly killed by a failing startup/liveness check before reaching Ready), and privileged containers. Each card shows the issue type, namespace, age, restart count, and a ready-to-run kubectl command. One click opens a full investigation.

Investigation — One click from detection to investigation. Every pod-level incident includes:

• OpsCart Assessment: what the pattern means and estimated investigation time, including whether the restart rate is accelerating
• Incident Timeline: an operational journal — first detected, restart milestones, severity changes, resolved/reoccurred — persisted across pod restarts
• Evidence: severity, first detected, restart count, state, age, owner
• Blast Radius: replicas down, sibling pod health, services routing to the workload, ingress exposure, namespace-wide health, and a customer-impact heuristic (internal vs. possible external traffic)
• Recommended Investigation: numbered steps with High / Medium / Low confidence and specific kubectl commands
• Recent Events: last 10 events filtered to this pod
• Related Resources: ConfigMaps, Secrets, PVCs referenced by the pod spec

Namespace-level findings (unprotected namespaces, idle namespaces) get their own dedicated view — a sample NetworkPolicy and namespace-scoped remediation steps, not a pod investigation that doesn't apply.

Incidents — The full system of record: every incident this cluster has seen, active and resolved. Search by name, namespace, or issue type; filter by severity and status; sorted and paginated. Resolved incidents stay visible with recovery time, so you can see what was broken last week — not just what's broken now.

Operational Insights

Security Posture — CIS Kubernetes Benchmark scoring. Failed controls shown first, risk breakdown by category, prioritized remediation actions.

Waste & Drift — Zombie pods, orphaned PVCs with storage size and age, zero-replica workloads, abandoned namespaces.

Cost Intelligence — Provider-aware worker-node estimates with namespace allocation, embedded Azure pricing, and optional AWS public pricing. See Cost Intelligence.

Platform

Operational Memory — OpsCart remembers what happened. A lightweight local database tracks cluster snapshots, incident lifecycle (detected → milestones → resolved → reopened) as an append-only event journal, and scan metadata. Powers trend arrows, sparklines, incident age, and the per-incident timeline. Backed by SQLite, persisted on a PVC that survives pod restarts and helm uninstall. Configurable retention (90 days by default) keeps the database from growing unbounded.

Authentication — Basic auth on by default with no disable path: environment variables, a Kubernetes Secret, or an auto-generated password logged at startup. For teams, front it with oauth2-proxy for Azure AD / Google / GitHub / OIDC — see First Login above.

Helm Chart — Full Helm chart with configurable values, PVC-backed persistence, read-only RBAC, and non-root security context. See the chart README for persistence options, minikube notes, and all values.

Agentless — Runs as a single container. No sidecars, no DaemonSets, and no node access. Core scanning needs no cloud credentials; optional AWS API pricing uses workload identity.

---

Security

Property	Detail
Base image	scratch — no OS, no shell, no package manager
User	Non-root (UID 65534)
Binary	CGO_ENABLED=0, statically compiled, -trimpath
CVE scan	0 vulnerabilities (Trivy)
Cluster permissions	Read-only ClusterRole (get, list only)
Mutations	None — never modifies cluster state
External calls	None — no telemetry, no phone-home
Authentication	Basic auth on by default, no disable path — env var, Secret, or auto-generated password

# Audit it yourself
trivy image ghcr.io/opscart/opscart-dashboard:latest
kubectl describe clusterrole opscart-dashboard

See docs/05-Security.md for the full threat model and security philosophy.

---

How It Compares

Tool	Primary Question
kubectl	What resources exist?
Lens / k9s	What is running right now?
Grafana	Are metrics within thresholds?
Kubecost	Where is money being spent?
OpsCart	What deserves attention first?

OpsCart is not a replacement for these tools. It is the triage layer that tells you which questions to ask of your observability stack.

---

Helm Configuration

See helm/opscart-watcher/README.md for the full values reference, persistence configuration, and environment-specific notes (minikube, multi-node, local images).

---

CLI Reference

go build -o opscart-scan ./cmd/opscart-scan

./opscart-scan emergency --cluster prod      # War Room from terminal
./opscart-scan security --cluster prod       # CIS scoring
./opscart-scan waste --cluster prod          # Waste detection
./opscart-scan cloud-costs --cluster prod    # Provider-aware cost analysis
./opscart-scan network --cluster prod        # Network policy gaps
./opscart-scan report --cluster prod         # HTML report

---

Coming Next

• Related incidents (cross-incident correlation — same namespace, shared config/secrets, correlated timing)
• Root cause confidence scoring (deterministic, evidence-based)
• Native OIDC authentication + namespace-scoped authorization (SubjectAccessReview-based)
• Slack and Teams notifications

---

Disclaimer

Awareness tool — not for formal compliance auditing. Use kube-bench for official CIS compliance. Cost figures are public-pricing estimates, not invoices; see Cost Intelligence.

---

Author: Shamsher Khan — opscart.com · IEEE Senior Member · DZone Core Member

License: MIT