engine

package
v0.50.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 13, 2026 License: Apache-2.0 Imports: 45 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	BottleneckIO      = "IO Starvation"
	BottleneckMemory  = "Memory Pressure"
	BottleneckCPU     = "CPU Contention"
	BottleneckNetwork = "Network Overload"
)

Variables

View Source 
var DomainPacks = map[string][]string{
	BottleneckCPU:     {"offcpu", "runqlat", "syscalldissect"},
	BottleneckIO:      {"iolatency", "wbstall"},
	BottleneckNetwork: {"tcprtt", "netthroughput", "sockio"},
	BottleneckMemory:  {"pgfault", "swapevict"},
}

DomainPacks maps RCA bottleneck names to probe packs.

View Source 
var Profiles = map[string]ThresholdProfile{
	"database": {
		"io.psi":            {Warn: 3, Crit: 10},
		"io.disk.latency":   {Warn: 10, Crit: 40},
		"io.disk.util":      {Warn: 60, Crit: 85},
		"mem.available.low": {Warn: 75, Crit: 90},
		"mem.swap.activity": {Warn: 1, Crit: 20},
	},
	"network": {
		"net.drops":       {Warn: 1, Crit: 50},
		"net.tcp.retrans": {Warn: 0.5, Crit: 3},
		"net.conntrack":   {Warn: 60, Crit: 85},
	},
	"mixed": {
		"io.psi":          {Warn: 4, Crit: 15},
		"io.disk.latency": {Warn: 15, Crit: 60},
		"io.disk.util":    {Warn: 65, Crit: 90},
		"net.drops":       {Warn: 1, Crit: 75},
		"net.tcp.retrans": {Warn: 0.8, Crit: 4},
		"net.conntrack":   {Warn: 65, Crit: 90},
	},
	"compute": {
		"cpu.psi":      {Warn: 3, Crit: 10},
		"cpu.runqueue": {Warn: 0.8, Crit: 1.5},
		"cpu.steal":    {Warn: 3, Crit: 10},
	},
	"gateway": {
		"net.conntrack": {Warn: 50, Crit: 80},
		"net.tcp.state": {Warn: 2000, Crit: 10000},
		"net.drops":     {Warn: 1, Crit: 30},
	},
}

Profiles defines role-based threshold override sets.

Functions

func AdaptiveThreshold 

func AdaptiveThreshold(db *AdaptiveThresholdDB, curr *model.Snapshot, metricName string, baseWarn, baseCrit float64) (warn, crit float64)

AdaptiveThreshold returns adaptive thresholds for a metric, auto-detecting workload. This is the convenience function RCA detectors should call.

func AnalyzeRCA 

func AnalyzeRCA(curr *model.Snapshot, rates *model.RateSnapshot, hist *History, peerIncidents map[string]*model.HostIncident, e *Engine) *model.AnalysisResult

AnalyzeRCA runs all bottleneck detectors and builds the full analysis result. peerIncidents provides cross-host correlation data (may be nil). AnalyzeRCA runs the RCA pass. NEXTGEN Phase 1 task 3: takes an optional *Engine for per-engine state. Pass nil for unit tests that don't need adaptive thresholds or causal-graph learning (the function falls back to the legacy package globals when e is nil — those will be removed in a follow-up plan once thresholdAdaptive is migrated).

func AppCulpritReason 

func AppCulpritReason(appType string, deepMetrics map[string]string, domain model.Domain) string

AppCulpritReason returns a human-readable reason when an app is the top process for a given domain. Returns "" when no app-specific reason applies.

func BuildCrossCorrelation 

func BuildCrossCorrelation(result *model.AnalysisResult, hist *History) []model.CrossCorrelation

BuildCrossCorrelation analyzes signal onset ordering across domains to detect cause-effect relationships. It checks predefined signal pairs and computes lead times from signal onsets tracked in History.

func BuildEntityGraph 

func BuildEntityGraph(snap *model.Snapshot) *model.EntityGraph

BuildEntityGraph constructs the host-local entity graph for one tick. NEXTGEN Phase 3 step 1 of the rollout: only process + cgroup nodes for now, with ownership edges. Future commits add service nodes, socket nodes, mount/container/pod nodes.

Ownership rules:

  • every process is owned by its cgroup (or by "host" if cgroup is empty or "/" — the kernel root)
  • every cgroup is owned by its parent cgroup, recursively up to "host" (which owns the root cgroup)

Cost: O(P + C) where P=processes, C=cgroups. On a 500-proc host the graph builds in ~100µs — within the engine's per-tick budget.

The graph is read-only after construction. Callers MUST NOT mutate Entities or the byID map after this returns.

func BuildNarrative 

func BuildNarrative(result *model.AnalysisResult, curr *model.Snapshot, rates *model.RateSnapshot) *model.Narrative

BuildNarrative produces a human-readable root cause narrative from analysis results.

func BuildProbabilisticDAG 

func BuildProbabilisticDAG(result *model.AnalysisResult, pcg *ProbabilisticCausalGraph, staticLearner ...*CausalLearner) *model.CausalDAG

BuildProbabilisticDAG builds a causal DAG using the probabilistic graph. It replaces the static rule table with learned inference when enough history exists (>= 50 observations). Otherwise falls back to static rules.

func BuildTemporalChain 

func BuildTemporalChain(result *model.AnalysisResult, hist *History) *model.TemporalChain

BuildTemporalChain constructs a temporal causality chain from the currently firing evidence, sorted by onset time (earliest first).

func BuildUSEChecklist 

func BuildUSEChecklist(snap *model.Snapshot, rates *model.RateSnapshot) []model.USECheck

BuildUSEChecklist generates Brendan Gregg's USE method checks from the current snapshot and rates: for every resource, check Utilization, Saturation, Errors.

func ComputeBlame 

func ComputeBlame(result *model.AnalysisResult, curr *model.Snapshot, rates *model.RateSnapshot) []model.BlameEntry

ComputeBlame identifies top offending processes/cgroups for the primary bottleneck.

func ComputeCapacity 

func ComputeCapacity(snap *model.Snapshot, rates *model.RateSnapshot) []model.Capacity

ComputeCapacity calculates headroom for key resources.

func ComputeFingerprint 

func ComputeFingerprint(e *model.Event, result *model.AnalysisResult) string

ComputeFingerprint generates a 16-char hex fingerprint for an incident. Hash: sha256(bottleneck|pattern_name|service_name)[:16].

func ComputeImpactScores 

func ComputeImpactScores(snap *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult) []model.ImpactScore

ComputeImpactScores calculates a composite impact score for each process. Weights: CPU=0.30, PSI=0.20, IO=0.20, Mem=0.20, Net=0.10. Newness penalty: +0.15 for processes started <60s ago.

func ComputeOwners 

func ComputeOwners(snap *model.Snapshot, rates *model.RateSnapshot) (cpu, mem, io, net []model.Owner)

ComputeOwners computes top resource consumers per subsystem.

func ComputeRates 

func ComputeRates(prev, curr *model.Snapshot) model.RateSnapshot

ComputeRates computes all rates between two snapshots.

func ComputeWarnings 

func ComputeWarnings(snap *model.Snapshot, rates *model.RateSnapshot) []model.Warning

ComputeWarnings detects early warning signals.

func DetectHiddenLatencyV2 

func DetectHiddenLatencyV2(curr *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult)

DetectHiddenLatencyV2 detects scheduler-level hidden latency using /proc/schedstat instead of rough heuristics.

func EnrichAppResourceShare 

func EnrichAppResourceShare(snap *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult, scores []model.ImpactScore)

EnrichAppResourceShare fills in the SRE "resource share" view for every detected app in snap.Global.Apps.Instances. Keeps each dimension separate (we never aggregate CPU+mem+IO into a composite percentage — that's an anti-pattern) and adds per-dimension ranks plus a bottleneck-share column when an incident is active.

Inputs:

  • snap: the live snapshot — for app list, NumCPUs, MemTotal.
  • rates: per-process + per-disk rates (gives us IO MB/s per PID and the worst disk's utilization baseline for IOPctOfBusiest).
  • result: the current AnalysisResult — if an incident is firing, its PrimaryBottleneck drives the BottleneckShare computation.
  • scores: ImpactScores the caller has already computed. Passing nil is fine; apps just won't show an impact number.

The function is a pure enrichment pass: it mutates the Share field on existing AppInstance entries and never adds/removes apps.

func EnrichNarrativeWithApps 

func EnrichNarrativeWithApps(narr *model.Narrative, result *model.AnalysisResult, apps []model.AppInstance)

EnrichNarrativeWithApps adds app-specific context to RCA narratives.

func EnrichSaturationBreakdown 

func EnrichSaturationBreakdown(gs *model.GoldenSignalSummary, curr *model.Snapshot, rates *model.RateSnapshot)

EnrichSaturationBreakdown populates the SaturationBreakdown field of a GoldenSignalSummary with per-component detail from raw snapshot data. This preserves all individual saturation signals that SaturationPct max() loses.

func EvaluateAppRCA 

func EvaluateAppRCA(snap *model.Snapshot, guardLevel int) []model.AppRCAFinding

EvaluateAppRCA runs the per-app rule library against the current snapshot and returns the findings. Skips entirely when guard level >= 1.

func FormatCorrelations 

func FormatCorrelations(correlations []IncidentCorrelation) []string

FormatCorrelations returns a human-readable summary of correlations.

func HoltExhaustionEvidence 

func HoltExhaustionEvidence(result *model.AnalysisResult, hist *History) []model.DegradationWarning

HoltExhaustionEvidence walks Forecaster ETAs and emits one DegradationWarning per resource that is forecast to hit critical within the warning horizon. This is the predictive complement to drift: drift catches "we're already in a bad place"; this catches "we're heading there fast".

func InjectConfigDriftEvidence 

func InjectConfigDriftEvidence(
	_ *model.EntityGraph,
	changes []model.SystemChange,
	measuredAt time.Time,
) []model.Fact

InjectConfigDriftEvidence converts a slice of model.SystemChange (only config_drift_* types are processed; others are silently skipped) into host-scoped model.Facts.

Each emitted Fact has:

  • Kind = FactKindConfigChange
  • Source = "config"
  • EntityID = "host" (config drift is always host-level)
  • Domain mapped from the change's Domain string (memory→DomainMemory, etc.)
  • Metric = the config key extracted from Detail
  • Value = 1.0 (presence signal — the magnitude is in the tags)
  • Confidence ≈ 0.6 (derived/interpreted per the FactConfidence rubric)
  • Tags: "key", "old", "new", "domain"

The graph parameter is accepted for API compatibility (future: graph-aware entity resolution). Currently unused because config drift is always host-scoped.

func InjectJournalEvidence 

func InjectJournalEvidence(
	graph *model.EntityGraph,
	serviceEntityID string,
	findings []journal.JournalFinding,
	measuredAt time.Time,
) []model.Fact

InjectJournalEvidence converts a slice of journal.JournalFinding into model.Facts scoped to serviceEntityID.

Entity scoping decision:

  • If graph != nil and serviceEntityID is present in the graph, the Fact's EntityID is set to serviceEntityID (preferred — tight scoping).
  • If graph != nil and serviceEntityID is ABSENT, EntityID falls back to "host" so the fact still lands in the verifier rather than being dropped.
  • If graph is nil, serviceEntityID is used as-is (caller takes responsibility for validity).

Findings are never dropped; the host fallback is a safety net for units not yet catalogued in the entity graph (e.g. transient kernel units).

Callers (P2.4 / P2.5) decide where to attach the returned Facts; this function only builds and returns them.

func NetHealthLevel 

func NetHealthLevel(snap *model.Snapshot, rates *model.RateSnapshot) string

NetHealthLevel returns the overall network health: "OK", "DEGRADED", or "CRITICAL".

func ProbeToEvidence 

func ProbeToEvidence(findings *ProbeFindings) []model.Evidence

ProbeToEvidence converts probe findings into synthetic RCA evidence objects. These can be injected into the RCA pipeline alongside /proc-derived evidence.

func QuantifyImpact 

func QuantifyImpact(result *model.AnalysisResult, snap *model.Snapshot, rates *model.RateSnapshot) string

QuantifyImpact generates a human-readable impact summary describing the real-world effect of the current bottleneck (e.g. blocked processes, latency increase, affected apps).

func ReadEventLog 

func ReadEventLog(path string) ([]model.Event, error)

ReadEventLog reads all events from a JSONL file.

func RunDaemon 

func RunDaemon(cfg DaemonConfig) error

RunDaemon runs xtop as a background collector, writing events to DataDir.

func ScrubCmdline 

func ScrubCmdline(s string) string

ScrubCmdline replaces credential-bearing tokens in a process command-line with "****". It is applied before cmdlines are shipped to the fleet hub so that passwords embedded in flags (e.g. mysql -pSecret) or environment variable assignments (AWS_SECRET_KEY=abc) are not transmitted in plaintext.

Safe to call on an empty string or a plain process name — no allocations occur when no patterns match.

func SelectProfile 

func SelectProfile(id *model.ServerIdentity) string

SelectProfile chooses the best threshold profile name for the given identity.

func SuggestActions 

func SuggestActions(result *model.AnalysisResult) []model.Action

SuggestActions generates actionable recommendations using data xtop already has. No shell commands — xtop IS the diagnostic tool.

func SummarizeForTrace 

func SummarizeForTrace(results []model.ProbeResult) string

SummarizeForTrace returns a compact one-liner per probe — used in the markdown rendering of the trace file.

func UpdateAppBaselines 

func UpdateAppBaselines(snap *model.Snapshot, rates *model.RateSnapshot, hist *History, frozen bool) []model.AppBehaviorAnomaly

UpdateAppBaselines feeds the per-app baseline trackers and emits anomalies. Returns the anomaly list to be attached to the analysis result.

frozen=true means "do not update the baseline, just check for anomalies" — used while an incident is Confirmed/Active so bad data isn't absorbed.

func UpdateDrift 

func UpdateDrift(result *model.AnalysisResult, hist *History, frozen bool) []model.DegradationWarning

UpdateDrift feeds the drift trackers and emits drift warnings for any metric whose long-window mean has diverged from its frozen reference.

frozen=true skips updates (used while a Confirmed incident is Active).

func UpdateSignalOnsets 

func UpdateSignalOnsets(hist *History, result *model.AnalysisResult)

UpdateSignalOnsets updates the signal onset tracking map in History. Call this each tick with the latest analysis result.

func WorstDiskGuardState 

func WorstDiskGuardState(rates []model.MountRate) string

WorstDiskGuardState returns the worst state across all mounts.

Types

type ActionType 

type ActionType string

ActionType identifies the kind of autopilot action. 

const (
	ActionThrottleCPU ActionType = "throttle_cpu"
	ActionIsolateProc ActionType = "isolate_process"
	ActionSetIONice   ActionType = "set_ionice"
)

type AdaptiveThresholdDB 

type AdaptiveThresholdDB struct {
	// contains filtered or unexported fields
}

AdaptiveThresholdDB stores learned thresholds per workload type.

func NewAdaptiveThresholdDB 

func NewAdaptiveThresholdDB(dataDir string) *AdaptiveThresholdDB

NewAdaptiveThresholdDB creates an adaptive threshold engine.

func (*AdaptiveThresholdDB) Close 

func (db *AdaptiveThresholdDB) Close()

Close stops the background save goroutine and flushes final state. Safe to call multiple times (idempotent).

func (*AdaptiveThresholdDB) Observe 

func (db *AdaptiveThresholdDB) Observe(wt WorkloadType, metricName string, value float64)

Observe records a metric sample for the given workload type.

func (*AdaptiveThresholdDB) Threshold 

func (db *AdaptiveThresholdDB) Threshold(wt WorkloadType, metricName string, baseWarn, baseCrit float64) (warn, crit float64)

Threshold returns the adaptive warn/crit thresholds for a metric. Falls back to baseWarn/baseCrit if insufficient data.

type AlertConfig 

type AlertConfig struct {
	Webhook          string
	Command          string
	Email            string
	SlackWebhook     string
	TelegramBotToken string
	TelegramChatID   string
}

AlertConfig defines alert destinations.

type AlertState 

type AlertState struct {

	// NoHysteresis disables the sustained-threshold requirement.
	// Use this for one-shot CLI/API mode where you want immediate health
	// reflection of the current score.
	NoHysteresis bool
	// contains filtered or unexported fields
}

AlertState implements a sustained-threshold alert state machine. State transitions (both escalation and de-escalation) require a sustained period (~15s wall-clock) at the new level. Critical evidence (OOM, disk ETA < 5m) can bypass the sustained requirement.

func NewAlertState 

func NewAlertState(intervalSec int) *AlertState

NewAlertState creates an AlertState calibrated for the given collection interval. The sustained requirement is computed to approximate sustainedWallClockSec.

func (*AlertState) Update 

func (as *AlertState) Update(health model.HealthLevel, score int, hasCritEvidence bool) model.HealthLevel

Update processes a new tick and returns the authoritative health level. health is the pre-computed health level (already respects v2 trust gate). score is the raw RCA score used for hysteresis thresholds. hasCritEvidence is true if instant-escalation evidence is present (e.g. OOM).

type AnomalyState 

type AnomalyState struct {
	// When the current primary bottleneck first crossed threshold
	PrimaryStart   time.Time
	PrimaryTrigger string // which signal first crossed
	PrimaryName    string // which bottleneck

	// When the current culprit became top
	CulpritStart time.Time
	CulpritName  string

	// Stability: when did health last become OK
	StableStart time.Time
}

AnomalyState tracks when bottlenecks first appeared.

type AppBaselineRecord 

type AppBaselineRecord struct {
	App        string
	Metric     string
	HourOfWeek int
	Count      int64
	Mean       float64
	M2         float64
}

AppBaselineRecord is the engine-local view of one persisted Welford bucket for application baselines. Mirrors store.AppBaselineRow.

type AppEvidenceInjector 

type AppEvidenceInjector struct{}

AppEvidenceInjector bridges app deep metrics into the RCA domain analyzers. It reads already-collected app metrics from the snapshot and emits EvidenceV2 entries that feed into domain scoring. Zero-cost when no apps are detected.

func NewAppEvidenceInjector 

func NewAppEvidenceInjector() *AppEvidenceInjector

NewAppEvidenceInjector creates an injector.

func (*AppEvidenceInjector) InjectCPUEvidence 

func (aei *AppEvidenceInjector) InjectCPUEvidence(db *AdaptiveThresholdDB, curr *model.Snapshot, r *model.RCAEntry)

InjectCPUEvidence scans app metrics and appends CPU-domain evidence.

func (*AppEvidenceInjector) InjectIOEvidence 

func (aei *AppEvidenceInjector) InjectIOEvidence(db *AdaptiveThresholdDB, curr *model.Snapshot, r *model.RCAEntry)

InjectIOEvidence scans app metrics and appends IO-domain evidence. Called from analyzeIO after system-level evidence is gathered.

func (*AppEvidenceInjector) InjectMemoryEvidence 

func (aei *AppEvidenceInjector) InjectMemoryEvidence(db *AdaptiveThresholdDB, curr *model.Snapshot, r *model.RCAEntry)

InjectMemoryEvidence scans app metrics and appends Memory-domain evidence.

func (*AppEvidenceInjector) InjectNetworkEvidence 

func (aei *AppEvidenceInjector) InjectNetworkEvidence(db *AdaptiveThresholdDB, curr *model.Snapshot, r *model.RCAEntry)

InjectNetworkEvidence scans app metrics and appends Network-domain evidence.

type Autopilot 

type Autopilot struct {
	// contains filtered or unexported fields
}

Autopilot manages safe automated remediation actions.

func NewAutopilot 

func NewAutopilot(cfg AutopilotConfig) *Autopilot

NewAutopilot creates a new autopilot instance.

func (*Autopilot) Actions 

func (a *Autopilot) Actions() []AutopilotAction

Actions returns a copy of all recorded actions.

func (*Autopilot) IsProtected 

func (a *Autopilot) IsProtected(comm string) bool

IsProtected returns true if the process is protected from autopilot actions.

func (*Autopilot) IsolateProcess 

func (a *Autopilot) IsolateProcess(pid int) error

IsolateProcess moves a process to an xtop-jail cgroup.

func (*Autopilot) Monitor 

func (a *Autopilot) Monitor(ticker Ticker, timeout time.Duration)

Monitor watches system health after actions and auto-rollbacks if things get worse.

func (*Autopilot) Rollback 

func (a *Autopilot) Rollback() error

Rollback reverses all autopilot actions in reverse order.

func (*Autopilot) SetIONice 

func (a *Autopilot) SetIONice(pid, class, level int) error

SetIONice changes the IO scheduling class/priority of a process.

func (*Autopilot) ThrottleCPU 

func (a *Autopilot) ThrottleCPU(cgroup string, quotaUs int) error

ThrottleCPU sets CPU quota on a cgroup to limit its CPU usage.

type AutopilotAction 

type AutopilotAction struct {
	Type      ActionType
	PID       int
	Comm      string
	Cgroup    string
	OldValue  string // original value for rollback
	NewValue  string
	Timestamp time.Time
}

AutopilotAction records an action taken by the autopilot for rollback.

type AutopilotConfig 

type AutopilotConfig struct {
	Enabled            bool     `json:"enabled"`
	AutoConfirm        bool     `json:"auto_confirm"` // skip confirmation prompts
	ProtectedServices  []string `json:"protected_services,omitempty"`
	MaxCPUQuotaUs      int      `json:"max_cpu_quota_us,omitempty"`     // min CPU quota to set (default 10000)
	MaxMemLimitMB      int      `json:"max_mem_limit_mb,omitempty"`     // min mem limit in MB
	RollbackTimeoutSec int      `json:"rollback_timeout_sec,omitempty"` // auto-rollback timeout (default 60)
}

AutopilotConfig configures the autopilot subsystem.

type BaselineTracker 

type BaselineTracker struct {
	// contains filtered or unexported fields
}

BaselineTracker maintains EWMA baselines for all evidence metrics.

func NewBaselineTracker 

func NewBaselineTracker(alpha float64) *BaselineTracker

NewBaselineTracker creates a tracker with the given EWMA decay factor. Alpha 0.01-0.05 is typical (smaller = slower adaptation).

func (*BaselineTracker) BaselineReadiness 

func (bt *BaselineTracker) BaselineReadiness() float64

BaselineReadiness returns the fraction of tracked metrics that have >= 30 samples (0.0 = all metrics warming up, 1.0 = fully ready).

func (*BaselineTracker) Get 

func (bt *BaselineTracker) Get(id string) (mean, stddev float64, ok bool)

Get returns the current baseline mean and stddev for a metric.

func (*BaselineTracker) IsAnomaly 

func (bt *BaselineTracker) IsAnomaly(id string, value, nSigma float64) bool

IsAnomaly returns true if value is more than nSigma standard deviations from baseline. Returns false if baseline is not warmed up. Uses a minimum stddev floor of minStddevFraction * |mean| to handle near-zero variance. The nSigma parameter is adjusted by distribution-aware thresholds (Item 15): if nSigma <= 0 the classified sigma for the metric ID is used automatically.

func (*BaselineTracker) IsWarmedUp 

func (bt *BaselineTracker) IsWarmedUp(id string) bool

IsWarmedUp returns true if enough samples have been collected for reliable baselines. Uses adaptive warmup: stops early when coefficient of variation stabilizes (Item 4).

func (*BaselineTracker) Update 

func (bt *BaselineTracker) Update(id string, value float64)

Update feeds a new value for a metric, updating its EWMA baseline.

func (*BaselineTracker) UpdateAll 

func (bt *BaselineTracker) UpdateAll(ids []string, values []float64)

UpdateAll feeds all fired evidence values into baselines.

func (*BaselineTracker) ZScore 

func (bt *BaselineTracker) ZScore(id string, value float64) float64

ZScore returns the z-score for a value against the baseline. Returns 0 if not warmed up.

type CUSUMTuning 

type CUSUMTuning struct {
	KMul float64
	HMul float64
}

CUSUMTuning controls change-point sensitivity. Expressed as multipliers of the metric's stddev so the same config works across vastly different metric magnitudes. KMul is the drift allowance; higher values ignore slow drifts. HMul is the alarm threshold in sigma units; higher values demand a larger cumulative deviation before firing.

Defaults are Page/Hinkley textbook values (K=0.5σ, H=4σ). Right-skewed metrics tolerate higher thresholds because their "normal" already includes occasional spikes — otherwise we'd churn baselines on every packet drop.

type CalibrationSummary 

type CalibrationSummary struct {
	Bottleneck     string    `json:"bottleneck"`
	TruePositives  int       `json:"tp"`
	FalsePositives int       `json:"fp"`
	Indeterminate  int       `json:"indet"`
	Factor         float64   `json:"factor"`
	Precision      float64   `json:"precision"`
	LastOutcomeAt  time.Time `json:"last_at"`
}

CalibrationSummary is what the post-mortem / fleet UI reads.

type CausalGraphEdge 

type CausalGraphEdge struct {
	From      string
	To        string
	PGiven    float64 // P(To | From) — learned from history
	PInverse  float64 // P(From | To) — for backward propagation
	Count     int     // co-occurrence count
	FirstSeen time.Time
	LastSeen  time.Time
}

CausalGraphEdge represents a directed edge with learned conditional probability.

type CausalGraphNode 

type CausalGraphNode struct {
	ID        string
	Domain    model.Domain
	Prior     float64 // base probability of this evidence firing
	Posterior float64 // updated after belief propagation
}

CausalGraphNode represents a node in the probabilistic causal graph.

type CausalLearner 

type CausalLearner struct {
	// contains filtered or unexported fields
}

CausalLearner tracks how often causal rules' predictions hold true and learns co-occurrence patterns from observed evidence.

func NewCausalLearner 

func NewCausalLearner() *CausalLearner

NewCausalLearner creates a causal learning tracker.

func (*CausalLearner) CoOccurrenceWeight 

func (cl *CausalLearner) CoOccurrenceWeight(from, to string) float64

CoOccurrenceWeight returns the learned co-occurrence weight between two evidence IDs. Returns 0 if insufficient data.

func (*CausalLearner) LearnedWeight 

func (cl *CausalLearner) LearnedWeight(rule string, hardcodedWeight float64, from, to string) float64

LearnedWeight returns a blended weight: 70% hardcoded + 30% observed. The observed component blends rule-level causal ordering with co-occurrence data. Returns the hardcoded weight if insufficient observations (<20).

func (*CausalLearner) Observe 

func (cl *CausalLearner) Observe(rule string, causeFired, effectFired bool, causeFirst bool)

Observe records whether a causal rule's prediction held for this tick.

func (*CausalLearner) ObserveCoOccurrence 

func (cl *CausalLearner) ObserveCoOccurrence(firedEvidence []string)

ObserveCoOccurrence records which evidence IDs fired together in a single tick. This builds a co-occurrence matrix used to supplement hardcoded causal weights.

type CausalRootCause 

type CausalRootCause struct {
	EvidenceID string
	Score      float64
	Posterior  float64
}

CausalRootCause is a ranked root-cause hypothesis.

func (CausalRootCause) String 

func (crc CausalRootCause) String() string

String returns a human-readable summary.

type ChangeDetector 

type ChangeDetector struct {
	// contains filtered or unexported fields
}

ChangeDetector tracks system changes between ticks.

func NewChangeDetector 

func NewChangeDetector() *ChangeDetector

NewChangeDetector creates a new change detector.

func (*ChangeDetector) DetectChanges 

func (cd *ChangeDetector) DetectChanges(snap *model.Snapshot) []model.SystemChange

DetectChanges compares current state to previous and returns changes.

type ConfidenceCalibrator 

type ConfidenceCalibrator struct {
	// contains filtered or unexported fields
}

ConfidenceCalibrator learns from incident outcomes whether the RCA's raw confidence score tends to be overstated or understated, per bottleneck, and multiplies future confidences by a correction factor.

Outcome labels:

  • true_positive: incident lasted ≥ 30 s AND peak_score ≥ 60 → a real event
  • false_positive: incident ended in < 8 s AND peak_score < 40 → noise
  • indeterminate: everything in between — excluded from calibration math

The correction factor is a smooth function of observed precision:

  • precision < 0.5 → factor 0.85 (down-weight: too many false positives)
  • precision ≥ 0.9 → factor 1.10 (up-weight: RCA is reliably right)
  • in between → linear interpolation

We cap the factor to [0.6, 1.2] so a single bad day can't silence the RCA and a lucky week can't push reported confidence past ceiling. At least 5 labelled incidents per bottleneck are required before any bias is applied.

Persisted to ~/.xtop/confidence-calibration.json so calibration survives restarts. Loaded on startup, saved after each outcome.

func NewConfidenceCalibrator 

func NewConfidenceCalibrator() *ConfidenceCalibrator

NewConfidenceCalibrator loads any existing calibration state from disk.

func (*ConfidenceCalibrator) ApplyTo 

func (c *ConfidenceCalibrator) ApplyTo(bottleneck string, rawConfidence int) int

ApplyTo returns the calibrated confidence given the raw engine output. The result is clamped to [0, 100] to match the model's nominal range.

func (*ConfidenceCalibrator) Factor 

func (c *ConfidenceCalibrator) Factor(bottleneck string) float64

Factor returns the multiplier to apply to a raw confidence score for the given bottleneck. Always returns 1.0 until at least 5 labelled incidents (TP+FP) have been recorded for that bottleneck.

func (*ConfidenceCalibrator) RecordOutcome 

func (c *ConfidenceCalibrator) RecordOutcome(bottleneck string, peakScore int, duration time.Duration)

RecordOutcome classifies a completed incident and updates calibration state. No-op for incidents without enough signal to classify (returns early on indeterminate outcomes so noise can't flatten the bias).

func (*ConfidenceCalibrator) Summary 

func (c *ConfidenceCalibrator) Summary() map[string]CalibrationSummary

Summary returns a human-readable snapshot for use in the post-mortem UI.

type ConfigBaselineRecord 

type ConfigBaselineRecord struct {
	Key       string
	Value     string
	Domain    string
	FirstSeen time.Time
	Acked     bool
}

ConfigBaselineRecord is the engine-local view of one persisted kernel-param baseline entry. Mirrors store.ConfigBaselineRow without importing store.

type ConfigDriftDetector 

type ConfigDriftDetector struct {
	// contains filtered or unexported fields
}

ConfigDriftDetector watches a curated set of high-impact config files and reports when any of them change. Drift events feed into the existing "Recent Activity" stream on the overview page, and — more importantly — the RCA narrative, so when an incident starts 8 minutes after a change to /etc/nginx/nginx.conf, the operator sees that immediately.

Design constraints:

  • Cheap: stat() on ~60 files every 30s, SHA256 only when mtime moves.
  • Resilient: missing files are fine; absence→presence counts as added, presence→absence as removed, hash change as modified.
  • Persistent: the baseline is saved to ~/.xtop/config-baseline.json so a restart doesn't re-flood "modified" events.
  • Bounded scope: NOT a general-purpose file watcher — only the explicit watchlist below (editable via $XTOP_CONFIG_WATCH).

func NewConfigDriftDetector 

func NewConfigDriftDetector() *ConfigDriftDetector

NewConfigDriftDetector creates a detector with the default watchlist.

func (*ConfigDriftDetector) RecentWithin 

func (d *ConfigDriftDetector) RecentWithin(ref time.Time, window time.Duration) []model.SystemChange

RecentWithin returns drift events that happened within `window` of `ref`. Used by RCA to correlate a config change with an incident that started shortly after.

func (*ConfigDriftDetector) Tick 

func (d *ConfigDriftDetector) Tick() []model.SystemChange

Tick runs a detection pass if scanEvery has elapsed since the last scan. Returns any newly-detected drift events (added / modified / removed). The detector also keeps a 6-hour buffer of events for RCA correlation — see RecentWithin.

type Correlator 

type Correlator struct {
	// contains filtered or unexported fields
}

Correlator tracks streaming Pearson correlations for pre-defined metric pairs.

func NewCorrelator 

func NewCorrelator() *Correlator

NewCorrelator creates a new streaming correlation tracker.

func (*Correlator) Add 

func (c *Correlator) Add(idA, idB string, valA, valB float64)

Add feeds a paired observation for two metrics.

func (*Correlator) BestLag 

func (c *Correlator) BestLag(idA, idB string) (lag int, r float64)

BestLag returns the lag (in samples) that maximizes cross-correlation for a given pair, along with the correlation at that lag. Positive lag means idA leads idB.

func (*Correlator) R 

func (c *Correlator) R(idA, idB string) float64

R returns the Pearson correlation coefficient for a pair. Returns 0 if insufficient data.

func (*Correlator) Samples 

func (c *Correlator) Samples(idA, idB string) int64

Samples returns how many effective observations exist for a pair.

func (*Correlator) TopCorrelations 

func (c *Correlator) TopCorrelations(minR float64, maxN int) []struct {
	A, B string
	R    float64
	N    int64
}

TopCorrelations returns the pairs with |R| > minR, sorted by |R| descending.

func (*Correlator) UpdateFromEvidence 

func (c *Correlator) UpdateFromEvidence(evidenceMap map[string]float64)

UpdateFromEvidence feeds current tick's evidence values into all tracked correlation pairs.

type DaemonAPIServer 

type DaemonAPIServer interface {
	// Update pushes the latest snapshot to the API snapshot provider.
	Update(snap *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult, scores []model.ImpactScore)
	// Serve starts the HTTP handler loop; returns http.ErrServerClosed on normal shutdown.
	Serve() error
	// Close shuts down the listener.
	Close() error
}

DaemonAPIServer abstracts the Unix-socket API server so daemon.go need not import the api package directly. The concrete implementation (api.Server + api.DaemonSnapshotProvider) is created in cmd/daemonwire.go.

type DaemonAggregateSample 

type DaemonAggregateSample struct {
	Health  string
	Score   int
	CPUBusy float64
	MemPct  float64
	IOPSI   float64
	TopPID  int
	TopComm string
}

DaemonAggregateSample is the engine-local view of a 10s aggregate sample. Mirrors store.AggregateSample.

type DaemonConfig 

type DaemonConfig struct {
	DataDir  string
	Interval time.Duration
	History  int
	Metrics  *MetricsStore
	Alerts   AlertConfig
	// Fleet carries optional hub-push settings. When Fleet.HubURL is set
	// and the engine detects it, the daemon pushes heartbeats + incidents
	// to the hub alongside its local event/summary writes.
	Fleet   model.FleetAgentConfig
	Version string // baked-in build version, passed to fleet heartbeats

	// Persistence is the optional sqlite-backed store. When nil, incident
	// storage and baseline persistence are disabled (in-memory only).
	// The concrete implementation lives in cmd/daemonwire.go to keep
	// modernc.org/sqlite out of the agent's import graph.
	Persistence DaemonStore

	// API is the optional Unix-socket API server. When nil, no API socket
	// is exposed. The concrete implementation lives in cmd/daemonwire.go.
	API DaemonAPIServer

	// ConfigDriftEnabled mirrors the --config-drift flag value. When false,
	// the kernel-parameter drift detector is disabled for this daemon run.
	// Defaults to false (safe), so cmd/root.go must set it explicitly from
	// the validated configDriftMode flag value.
	ConfigDriftEnabled bool
}

DaemonConfig holds daemon-specific configuration.

type DaemonStore 

type DaemonStore interface {
	// Baseline Welford state (app performance baselines)
	SaveAppBaselines(rows []AppBaselineRecord) error
	LoadAppBaselines() ([]AppBaselineRecord, error)

	// Drift tracker Welford state
	SaveDriftTrackers(rows []DriftTrackerRecord) error
	LoadDriftTrackers() ([]DriftTrackerRecord, error)

	// Kernel-param config baseline (P4.2 / P4.3)
	SaveConfigBaseline(rows []ConfigBaselineRecord) error
	LoadConfigBaseline() ([]ConfigBaselineRecord, error)

	// Incident tracking
	InsertIncident(e model.Event, fingerprint string, offenders []model.ImpactScore) error
	UpdateIncident(id string, e model.Event) error
	InsertAggregate(ts time.Time, agg DaemonAggregateSample) error
	UpsertFingerprint(fp IncidentFingerprint) error
	Prune(cutoff time.Time) (int, error)
}

DaemonStore is the full persistence interface used by the engine daemon path. Implementations live in cmd/ (never in engine/ or agent-reachable packages). Passing nil to any engine function that accepts DaemonStore is always safe — callers guard with `if s != nil`.

type DriftTrackerRecord 

type DriftTrackerRecord struct {
	Metric string
	Window string // "short" | "long" | "ref"
	Count  int64
	Mean   float64
	M2     float64
	RefSet bool
}

DriftTrackerRecord is the engine-local view of one persisted Welford bucket within a drift window. Mirrors store.DriftTrackerRow.

type Engine 

type Engine struct {
	History *History
	Smart   *collector.SMARTCollector

	Sentinel    *bpf.SentinelManager
	Watchdog    *WatchdogTrigger
	SecWatchdog *bpf.SecWatchdog // security deep-inspection watchdog
	MultiRes    *MultiResBuffer  // multi-resolution time series (nil if unused)
	SLOPolicies []SLOPolicy      // SLO policies from config/flags
	Autopilot   *Autopilot       // autopilot subsystem (nil if disabled)
	// contains filtered or unexported fields
}

Engine orchestrates collection, analysis, and scoring.

func NewEngine 

func NewEngine(historySize, intervalSec int) *Engine

NewEngine creates a new engine with all collectors registered. intervalSec is the collection interval used to calibrate alert thresholds.

func NewEngineMode 

func NewEngineMode(historySize, intervalSec int, mode collector.Mode) *Engine

NewEngineMode constructs an engine in the requested collector mode. Lean mode (used by daemon + fleet-hub agents) registers a small fixed set of essential collectors and skips the heavy ones — apps deep scanning, profiler audits, the cgroup tree walk, the eBPF sentinel program, all of the runtime detectors. The RCA engine downstream degrades gracefully on missing signals.

On top of the mode's baseline list, ~/.xtop/modules.json (managed by the v0.45 module toggle subcommand + UI) further filters which Optional and Heavy modules actually run. Essential collectors always run; the module config can never disable them.

func (*Engine) ApplyRCAThresholds 

func (e *Engine) ApplyRCAThresholds(t config.RCAThresholds)

ApplyRCAThresholds resolves config thresholds (zero → const default) and stores them on the engine. Call once after NewEngine, before first Tick.

func (*Engine) ArmTraceNext 

func (e *Engine) ArmTraceNext()

ArmTraceNext arms the trace armer to dump on the next tick. CLI helper.

func (*Engine) ArmTraceOnConfirmed 

func (e *Engine) ArmTraceOnConfirmed()

ArmTraceOnConfirmed arms continuous Confirmed-transition dumps. CLI helper.

func (*Engine) AttachFleetClient 

func (e *Engine) AttachFleetClient(fc *FleetClient, version string)

AttachFleetClient wires a fleet push client into the engine. Called once at startup by cmd/root.go when --fleet-hub is configured. Hostname and version are captured once since they never change at runtime.

func (*Engine) Base 

func (e *Engine) Base() *Engine

Base returns itself for the default engine ticker.

func (*Engine) Calibrator 

func (e *Engine) Calibrator() *ConfidenceCalibrator

Calibrator exposes the confidence calibration table so the UI / post-mortem can show "this bottleneck has precision X over N past incidents." Returns nil on engines that were constructed without calibration (tests).

func (*Engine) Close 

func (e *Engine) Close()

Close shuts down all engine resources (sentinel probes, security watchdog, collector connections). Cleanup runs in parallel with a 2-second hard timeout so quit is never slow.

func (*Engine) CorpusStats 

func (e *Engine) CorpusStats() IncidentCorpusStats

CorpusStats returns the current capture stats.

func (*Engine) GetPeerIncidents 

func (e *Engine) GetPeerIncidents() map[string]*model.HostIncident

GetPeerIncidents returns a snapshot of all known peer host incidents.

func (*Engine) Guard 

func (e *Engine) Guard() *ResourceGuard

Guard exposes the live ResourceGuard so the status line and tests can inspect it. Nil until the first Tick constructs it (needs CPU count).

func (*Engine) LoadBaselineState 

func (e *Engine) LoadBaselineState(s DaemonStore) error

LoadBaselineState reads previously persisted Welford state and seeds the in-memory stores. Engine calls this once on startup, before the first Tick. Safe with nil store (no-op).

func (*Engine) ReportPeerIncident 

func (e *Engine) ReportPeerIncident(hostID string, inc *model.HostIncident)

ReportPeerIncident records the latest incident from a peer host for cross-host correlation.

func (*Engine) Runbooks 

func (e *Engine) Runbooks() *RunbookLibrary

Runbooks exposes the in-memory runbook library so the UI can fetch the full markdown body for a matched runbook (which isn't carried on the AnalysisResult to keep it small).

func (*Engine) SaveBaselineState 

func (e *Engine) SaveBaselineState(s DaemonStore) error

SaveBaselineState writes the in-memory Welford state for app baselines, drift trackers, and pending kernel-param baselines to the supplied store. Engine calls this periodically and on Close. Safe to call with nil store (no-op).

func (*Engine) SetConfigDriftEnabled 

func (e *Engine) SetConfigDriftEnabled(enabled bool)

SetConfigDriftEnabled enables or disables the kernel-parameter drift detector (--config-drift flag). Default is true (enabled).

func (*Engine) SetNoHysteresis 

func (e *Engine) SetNoHysteresis(v bool)

SetNoHysteresis disables the sustained-threshold alert state machine. When true, health level reflects the instantaneous score without requiring consecutive ticks. Use this for one-shot CLI/API mode.

func (*Engine) Tick 

func (e *Engine) Tick() (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)

Tick performs one collection + analysis cycle. Returns the snapshot, computed rates, and full analysis result. Serialized via tickMu to prevent concurrent collection when ticks overlap.

type EventDetector 

type EventDetector struct {
	// contains filtered or unexported fields
}

EventDetector tracks health state transitions and emits events.

func NewEventDetector 

func NewEventDetector() *EventDetector

NewEventDetector creates a new detector with default debounce of 3 ticks.

func (*EventDetector) ActiveEvent 

func (d *EventDetector) ActiveEvent() *model.Event

ActiveEvent returns a copy of the current active event, or nil.

func (*EventDetector) AllEvents 

func (d *EventDetector) AllEvents() (active *model.Event, completed []model.Event)

AllEvents returns active (if any) + completed events for display. Returns a copy of the active event to avoid data races.

func (*EventDetector) Events 

func (d *EventDetector) Events() []model.Event

Events returns completed events in reverse chronological order.

func (*EventDetector) LoadEvents 

func (d *EventDetector) LoadEvents(events []model.Event)

LoadEvents adds externally loaded events (e.g., from daemon log).

func (*EventDetector) Process 

func (d *EventDetector) Process(snap *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult)

Process is called every tick with the current analysis result. It detects health transitions and manages events.

type EventLogWriter 

type EventLogWriter struct {
	// contains filtered or unexported fields
}

EventLogWriter appends events to a JSONL file with size-based rotation.

func NewEventLogWriter 

func NewEventLogWriter(path string) *EventLogWriter

NewEventLogWriter creates a writer for the given path.

func (*EventLogWriter) Write 

func (w *EventLogWriter) Write(e model.Event) error

Write appends an event to the log file, rotating if it exceeds maxSize.

type FastPulse 

type FastPulse struct {
	// contains filtered or unexported fields
}

FastPulse is a sub-second sampler for PSI. It complements xtop's tick-based collector (default 3s) by tracking the wall-clock onset of pressure events at ~500ms resolution. The output is consumed by stampSustainedDurations to give per-evidence SustainedForSec values that the trust gates can rely on.

Design: one goroutine, one read loop, no allocations on the hot path beyond the parser. The cost target is <1ms per pulse; on a healthy host this whole loop costs ~30ms of CPU per minute.

Disabled if XTOP_FASTPULSE=0. We never ship a no-op stub on non-Linux — PSI is Linux-only, and xtop is Linux-only, so the build tag is enough.

func NewFastPulse 

func NewFastPulse(intervalMs int) *FastPulse

NewFastPulse builds a stopped FastPulse. Start() launches the loop. intervalMs <= 0 defaults to 500ms.

func (*FastPulse) Latest 

func (fp *FastPulse) Latest(id string) (val float64, age time.Duration, ok bool)

Latest returns the most recent sample for an id, with the age since it was taken. ok=false if never observed.

func (*FastPulse) Start 

func (fp *FastPulse) Start()

Start launches the pulse goroutine. Idempotent.

func (*FastPulse) Stop 

func (fp *FastPulse) Stop()

Stop signals the goroutine to exit. Safe to call multiple times.

Race-safety: Stop CLOSES the existing quit channel and replaces it with a fresh one for any future Start. The goroutine reads its channel via a captured local — never the field — so the close → re-create sequence is safe even if another Start follows.

func (*FastPulse) SustainedAbove 

func (fp *FastPulse) SustainedAbove(id string) (time.Duration, bool)

SustainedAbove returns how long the signal has been continuously above its configured threshold, with sub-second precision. The second return is false if there is no current streak (signal is below threshold or never seen).

type FleetClient 

type FleetClient struct {
	// contains filtered or unexported fields
}

func NewFleetClient 

func NewFleetClient(cfg model.FleetAgentConfig) *FleetClient

NewFleetClient creates a client. Returns nil if cfg.HubURL is empty.

func (*FleetClient) AgentID 

func (fc *FleetClient) AgentID() string

AgentID returns the stable agent identifier.

func (*FleetClient) Close 

func (fc *FleetClient) Close()

Close stops the worker goroutine and flushes any pending messages to disk.

func (*FleetClient) HubURL 

func (fc *FleetClient) HubURL() string

HubURL returns the hub URL the client is pushing to. Used for logging.

func (*FleetClient) Observe 

func (fc *FleetClient) Observe(snap *model.Snapshot, result *model.AnalysisResult, hostname, agentVersion string)

Observe is called by the engine every tick with the current snapshot/result. It builds + queues a heartbeat, and emits incident records on state changes. This call must be non-blocking — any IO happens in the worker.

type FleetQuality 

type FleetQuality struct {
	MinPeakScore     int           // below this → never emit (default 30)
	MinConfidence    int           // below this → never emit (default 40)
	MinStartTicks    int           // consecutive bad ticks before IncidentStarted (default 3)
	MinEscalationGap time.Duration // min time between signature-flip escalations (default 15s)
}

FleetClient pushes heartbeats and incidents to an xtop hub. FleetQuality is the per-agent incident quality gate. It exists because the raw RCA signal flaps near the OK/Degraded boundary — a box briefly over threshold for one tick produces a pure-noise "incident" that's useless to operators and floods the hub's Postgres. The defaults below suppress 0%-score / 0%-confidence flaps; operators can tighten them further via XTOP_FLEET_QUALITY_* env vars.

type ForecastRegime 

type ForecastRegime int

ForecastRegime describes the current behavior of a metric. 

const (
	RegimeStable   ForecastRegime = iota // low volatility, predictable
	RegimeTrending                       // consistent directional movement
	RegimeNoisy                          // high volatility, unpredictable
	RegimeSpiky                          // mostly flat with occasional spikes
)

func (ForecastRegime) String 

func (r ForecastRegime) String() string

type GuardAdvice 

type GuardAdvice struct {
	Level         int     // 0..3
	Reason        string  // short, human-readable
	IntervalSec   int     // suggested tick interval
	SkipAppDeep   bool    // skip expensive app deep-metric probes
	SkipLogTailer bool    // skip app-log correlation (file IO)
	SkipTraces    bool    // skip OTel trace correlator
	SkipWatchdog  bool    // skip auto-triggering eBPF watchdogs
	SkipDeepScan  bool    // pause the full-FS polite scanner
	OwnCPUPct     float64 // our own CPU%, for display
	HostLoadRatio float64 // load/NumCPUs, for display
}

GuardAdvice is the per-tick output consumed by Engine.Tick. Everything the engine needs to decide what to skip lives on this struct.

type History 

type History struct {

	// Statistical intelligence
	Baselines      *BaselineTracker
	ZScores        *ZScoreTracker
	Correlator     *Correlator
	Forecaster     *HoltForecaster
	Seasonal       *SeasonalTracker
	CausalLearner  *CausalLearner
	ProcessHistory *ProcessHistory

	// FastPulse provides sub-second PSI onset tracking. Optional; nil disables.
	// Set by NewEngine when XTOP_FASTPULSE != "0".
	FastPulse *FastPulse
	// contains filtered or unexported fields
}

History is a ring buffer of snapshots and rates for trend detection.

func NewHistory 

func NewHistory(capacity, intervalSec int) *History

NewHistory creates a ring buffer with the given capacity. intervalSec is the collection interval, used to calibrate alert thresholds.

func (*History) FlushOlderThan 

func (h *History) FlushOlderThan(keep int)

FlushOlderThan keeps only the N most recent snapshots and zeros the rest of the ring so Go's GC can reclaim the underlying memory. Called by the Guardian when heap usage exceeds the hard budget — preserves enough state for RCA (just needs current+previous tick) while dropping the long-tail history that's only useful for TUI scrollback.

func (*History) Get 

func (h *History) Get(i int) *model.Snapshot

Get returns a copy of the snapshot at position i (0 = oldest in buffer).

func (*History) GetRate 

func (h *History) GetRate(i int) *model.RateSnapshot

GetRate returns a copy of the rate snapshot at position i (0 = oldest in buffer).

func (*History) Latest 

func (h *History) Latest() *model.Snapshot

Latest returns a copy of the most recent snapshot.

func (*History) Len 

func (h *History) Len() int

Len returns the number of snapshots stored.

func (*History) Previous 

func (h *History) Previous() *model.Snapshot

Previous returns a copy of the snapshot before the most recent one.

func (*History) Push 

func (h *History) Push(snap model.Snapshot)

Push adds a snapshot to the ring buffer.

func (*History) PushRate 

func (h *History) PushRate(rate model.RateSnapshot)

PushRate stores a rate snapshot at the same position as the latest Push.

func (*History) SetNoHysteresis 

func (h *History) SetNoHysteresis(v bool)

SetNoHysteresis disables the sustained-threshold alert state machine.

type HoltForecaster 

type HoltForecaster struct {
	// contains filtered or unexported fields
}

HoltForecaster provides Holt-Winters triple exponential smoothing forecasting with 24-hour seasonal cycle and accuracy tracking.

func NewHoltForecaster 

func NewHoltForecaster(alpha, beta float64) *HoltForecaster

NewHoltForecaster creates a Holt forecaster with given alpha (level) and beta (trend) params. These values seed each new per-metric state; per-metric alpha/beta then evolve independently.

func (*HoltForecaster) ETAToThreshold 

func (hf *HoltForecaster) ETAToThreshold(id string, threshold float64, maxSteps int) float64

ETAToThreshold returns how many steps until the metric crosses a threshold. Returns -1 if trend is not heading toward threshold, 0 if insufficient data. Caps at maxSteps.

func (*HoltForecaster) Forecast 

func (hf *HoltForecaster) Forecast(id string, h int) float64

Forecast returns the predicted value h steps ahead. Returns 0 if insufficient data.

func (*HoltForecaster) ForecastCI 

func (hf *HoltForecaster) ForecastCI(id string, h int, futureHour int) (mean, lower, upper float64)

ForecastCI returns forecast with 95% confidence interval bounds. futureHour is the hour (0-23) of the forecast target; pass -1 to skip seasonality.

func (*HoltForecaster) ForecastSeasonal 

func (hf *HoltForecaster) ForecastSeasonal(id string, h int, futureHour int) float64

ForecastSeasonal returns the predicted value h steps ahead with seasonal adjustment. futureHour is the hour (0-23) of the forecast target.

func (*HoltForecaster) RMSE 

func (hf *HoltForecaster) RMSE(id string) float64

RMSE returns the root mean squared error of past forecasts for a metric.

func (*HoltForecaster) Regime 

func (hf *HoltForecaster) Regime(id string) ForecastRegime

Regime returns the current forecast regime for a metric.

func (*HoltForecaster) Trend 

func (hf *HoltForecaster) Trend(id string) float64

Trend returns the current trend rate (units per step).

func (*HoltForecaster) Update 

func (hf *HoltForecaster) Update(id string, value float64)

Update feeds a new observation for a metric.

func (*HoltForecaster) UpdateWithHour 

func (hf *HoltForecaster) UpdateWithHour(id string, value float64, hour int)

UpdateWithHour feeds a new observation for a metric with the current hour (0-23) for seasonal adjustment. Pass hour < 0 to skip seasonality.

type HostTopology 

type HostTopology struct {
	HostID   string
	Role     TopologyRole
	Tier     int      // 0 = edge/lb, 1 = web, 2 = app, 3 = data
	Region   string   // AWS region, datacenter, etc.
	Zone     string   // AZ, rack, etc.
	Peers    []string // directly connected host IDs
	Services []string // services running on this host
}

HostTopology describes a host's position and connections in the infra graph.

type IOLatEntry 

type IOLatEntry struct {
	Device  string
	P50Ms   float64
	P95Ms   float64
	P99Ms   float64
	UtilPct float64
}

IOLatEntry holds block IO latency for one device.

type IncidentCorpusStats 

type IncidentCorpusStats struct {
	Dir         string `json:"dir"`
	WrittenLife int    `json:"written_lifetime"`
}

IncidentCorpusStats reports how many frames this engine has captured. Useful for operators wanting to know "is xtop building me a corpus?".

type IncidentCorrelation 

type IncidentCorrelation struct {
	SourceHost     string
	TargetHost     string
	SourceIncident *model.HostIncident
	TargetIncident *model.HostIncident
	Distance       int     // graph hops
	Likelihood     float64 // 0-1, probability that source caused target
	Direction      string  // "upstream", "downstream", "peer", "unknown"
	TimeDeltaSec   float64
}

IncidentCorrelation is a correlated incident between two hosts.

type IncidentFingerprint 

type IncidentFingerprint struct {
	FP          string
	FirstSeen   time.Time
	LastSeen    time.Time
	Count       int
	AvgDuration int
	SymptomType string
	RootClass   string
	TopOffender string
}

IncidentFingerprint is the engine-local view of an incident fingerprint row. Mirrors store.Fingerprint.

type IncidentRecorder 

type IncidentRecorder struct {
	// contains filtered or unexported fields
}

IncidentRecorder tracks ongoing incidents and persists completed ones to disk.

func NewIncidentRecorder 

func NewIncidentRecorder() *IncidentRecorder

NewIncidentRecorder creates a recorder that persists to ~/.xtop/rca-history.jsonl.

func (*IncidentRecorder) Active 

func (r *IncidentRecorder) Active() *RCAIncident

Active returns a copy of the currently active incident, or nil if the system is healthy. Used by the confidence calibrator to detect incident completions at the moment health flips back to OK.

func (*IncidentRecorder) DiffAgainstHistory 

func (r *IncidentRecorder) DiffAgainstHistory(result *model.AnalysisResult) *model.IncidentDiff

DiffAgainstHistory compares the current RCA result against the last N similar incidents (same signature) and returns a structured IncidentDiff.

This is the mechanical heart of the "this vs last N similar" feature: the UI/narrative layer reads the returned struct to surface things like "score is 15 points worse than usual" or "swap_churn is firing this time but wasn't in the last 4 incidents — check why we're swapping now."

Returns nil when there are no prior matches (first occurrence, or history file not populated yet).

func (*IncidentRecorder) FindSimilar 

func (r *IncidentRecorder) FindSimilar(result *model.AnalysisResult) []RCAIncident

FindSimilar returns past incidents matching the current pattern/signature. Used to enrich the current RCA with historical context.

func (*IncidentRecorder) HistoryContext 

func (r *IncidentRecorder) HistoryContext(result *model.AnalysisResult) string

HistoryContext builds a human-readable context string from similar past incidents. Used to enrich current RCA narratives with "this happened N times before" insights.

func (*IncidentRecorder) HistoryStats 

func (r *IncidentRecorder) HistoryStats() (total int, last24h int, top3Bottlenecks []string)

HistoryStats returns aggregate statistics for the status line.

func (*IncidentRecorder) Record 

func (r *IncidentRecorder) Record(result *model.AnalysisResult) *RCAIncident

Record processes the current RCA result, tracking incident start/end with the Suspected → Confirmed → Resolved lifecycle.

Promotion rules:

  • First non-OK tick: start Suspected (in-memory only).
  • Subsequent ticks: if confirmedTrustGate passes (sustained evidence + diversity), promote Suspected → Confirmed and stamp ConfirmedAt.
  • Health returns to OK: if state was ever Confirmed, persist to JSONL. Pure-Suspected episodes are dropped as noise — this is the primary false-positive guard.

Returns the currently active incident (nil if healthy or only Suspected and caller wants to hide it). Suspected incidents are returned so the UI can optionally show "investigating…" status without alerting.

type IncidentState 

type IncidentState string

IncidentState tracks the lifecycle of an in-flight incident. 

Suspected: pressure detected this tick but not yet confirmed (in-memory only,
           never persisted to JSONL or surfaced as "incident" in the UI status).
Confirmed: passed confirmedTrustGate (sustained evidence + diversity gate).
           First time we treat this as a real incident; persists on close.
Resolved:  health returned to OK. Persists to JSONL only if it was ever
           Confirmed; pure-Suspected episodes are discarded as noise.

const (
	IncidentSuspected IncidentState = "suspected"
	IncidentConfirmed IncidentState = "confirmed"
	IncidentResolved  IncidentState = "resolved"
)

type JournalQueryFn 

type JournalQueryFn func(ctx context.Context, unit string, since time.Time) ([]journal.Entry, error)

JournalQueryFn is the injectable journalctl query function used by the Tier-1 journal pipeline. Default is journal.Query (which is a no-op stub on non-Linux via collector/journal/query_stub.go).

type LockEntry 

type LockEntry struct {
	PID      int
	Comm     string
	WaitPct  float64
	LockType string
}

LockEntry holds lock contention for one process.

type LogTailer 

type LogTailer struct {
	// contains filtered or unexported fields
}

LogTailer correlates an active incident's RCA verdict with the application's own log output. When RCA points at mysqld, we show mysql's slow-query output; when it points at nginx, we surface the latest "upstream timed out" errors. This is the single highest-signal cross-reference an operator can get without switching terminals.

Design:

  • Watchlist maps app name → list of candidate log paths + severity regex.
  • Tailer reads only the last ~64 KiB of each file (seek-from-end) per scan.
  • Scans are rate-limited to once every 10 s and bounded to 25 ms total wall-clock per tick, so a badly-behaved log never slows collection.
  • Systemd journal fallback: when no path matches, we fall back to `journalctl -u <culprit> --since=2m --no-pager --lines=200`.

Privacy: we deliberately don't fingerprint log content or send log lines to the fleet hub. Excerpts live only on the local AnalysisResult.

func NewLogTailer 

func NewLogTailer() *LogTailer

NewLogTailer returns a tailer pre-loaded with the built-in watchlist.

func (*LogTailer) Flush 

func (t *LogTailer) Flush()

Flush drops the rate-limit cache so the next Observe re-scans from scratch. Called by the Guardian when memory pressure forces a cache purge. Cheap (map-clear) and safe — worst case is one extra scan on the next active-incident tick.

func (*LogTailer) Observe 

func (t *LogTailer) Observe(result *model.AnalysisResult) []model.LogExcerpt

Observe returns a list of log excerpts relevant to the current incident, or nil when there's nothing worth attaching. Must be cheap — it is called on every Tick while an incident is active.

type MetricDistribution 

type MetricDistribution int

MetricDistribution classifies the expected statistical distribution of a metric (Item 15). 

const (
	DistNormal      MetricDistribution = iota
	DistRightSkewed                    // retrans, drops, OOM kills — mostly 0, occasional spikes
	DistBimodal                        // PSI — either 0 or significant
)

type MetricsStore 

type MetricsStore struct {
	// contains filtered or unexported fields
}

MetricsStore holds the latest snapshot for exporters.

func NewMetricsStore 

func NewMetricsStore() *MetricsStore

NewMetricsStore creates a new store.

func (*MetricsStore) Handler 

func (s *MetricsStore) Handler() http.Handler

Handler exposes Prometheus metrics for the latest sample.

func (*MetricsStore) Snapshot 

func (s *MetricsStore) Snapshot() (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult, time.Time)

Snapshot returns the latest stored sample.

func (*MetricsStore) Update 

func (s *MetricsStore) Update(snap *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult)

Update stores the latest sample.

type MountGrowthTracker 

type MountGrowthTracker struct {
	// contains filtered or unexported fields
}

MountGrowthTracker smooths filesystem growth rates using EWMA.

func NewMountGrowthTracker 

func NewMountGrowthTracker() *MountGrowthTracker

NewMountGrowthTracker creates a new tracker.

func (*MountGrowthTracker) Smooth 

func (t *MountGrowthTracker) Smooth(rates []model.MountRate)

Smooth applies EWMA to growth rates and recomputes ETA and state.

type MultiResBuffer 

type MultiResBuffer struct {
	HiRes  []TimeSeries1s // 1-sample resolution, last 5 min (300 samples at 1s)
	MidRes []TimeSeries1m // 1-minute averages, last 1 hour
	LowRes []TimeSeries1h // 1-hour averages, last 24 hours
	// contains filtered or unexported fields
}

MultiResBuffer maintains metrics at 3 resolutions for trend analysis.

func NewMultiResBuffer 

func NewMultiResBuffer() *MultiResBuffer

NewMultiResBuffer creates a new multi-resolution buffer.

func (*MultiResBuffer) HiResWindow 

func (b *MultiResBuffer) HiResWindow(n int) []TimeSeries1s

HiResWindow returns the most recent n high-resolution samples.

func (*MultiResBuffer) LowResWindow 

func (b *MultiResBuffer) LowResWindow(n int) []TimeSeries1h

LowResWindow returns the most recent n 1-hour samples.

func (*MultiResBuffer) MidResWindow 

func (b *MultiResBuffer) MidResWindow(n int) []TimeSeries1m

MidResWindow returns the most recent n 1-minute samples.

func (*MultiResBuffer) PushHiRes 

func (b *MultiResBuffer) PushHiRes(ts TimeSeries1s)

PushHiRes adds a high-resolution sample and triggers aggregation.

type NetThroughputEntry 

type NetThroughputEntry struct {
	PID   int
	Comm  string
	TxMBs float64
	RxMBs float64
}

NetThroughputEntry holds per-process TCP throughput.

type Notifier 

type Notifier struct {
	// contains filtered or unexported fields
}

Notifier sends alert notifications.

func NewNotifier 

func NewNotifier(cfg AlertConfig) *Notifier

NewNotifier creates a notifier.

func (*Notifier) Close 

func (n *Notifier) Close()

Close shuts down the notifier's worker goroutine by closing the job queue. It is safe to call Close more than once; subsequent calls are no-ops. Close should be called only after all Notify callers have stopped, because there is a narrow TOCTOU window between the closed.Load() check in Notify and the channel send: if Close races with an in-flight Notify the send is guarded by a recover() to prevent a panic on the closed channel.

func (*Notifier) Enabled 

func (n *Notifier) Enabled() bool

Enabled returns true if any alert destination is configured.

func (*Notifier) Notify 

func (n *Notifier) Notify(event string, payload interface{})

Notify sends an alert event asynchronously.

func (*Notifier) SendFormatted 

func (n *Notifier) SendFormatted(event, subject, text string, payload interface{})

SendFormatted dispatches a formatted alert to all configured channels.

type OffCPUEntry 

type OffCPUEntry struct {
	PID     int
	Comm    string
	WaitPct float64
	Reason  string
}

OffCPUEntry holds off-CPU time attribution for one process.

type ParamDriftDetector 

type ParamDriftDetector struct {
	// contains filtered or unexported fields
}

ParamDriftDetector compares live kernel-parameter snapshots against a persisted baseline and emits model.SystemChange events for drifted values.

Thread-safety: not concurrent-safe by design — the engine calls Detect synchronously on the tick goroutine. Wrap with a mutex if that changes.

func NewParamDriftDetector 

func NewParamDriftDetector(rows []ConfigBaselineRecord) *ParamDriftDetector

NewParamDriftDetector initialises a detector from the persisted baseline records (typically loaded via DaemonStore.LoadConfigBaseline). Passing nil or an empty slice starts with an empty baseline — every first-seen key will be collected as a newBaseline entry on the first Detect call.

func (*ParamDriftDetector) Detect 

func (d *ParamDriftDetector) Detect(live map[string]string, now time.Time) (changes []model.SystemChange, newBaselines []ConfigBaselineRecord)

Detect compares live key→value pairs against the baseline and returns:

  • changes: drift SystemChanges for keys whose value differs from baseline. Type is "config_drift_<domain>"; Detail is "key: old → new".
  • newBaselines: entries for keys not yet in the baseline (first-ever-seen). The caller should persist these via DaemonStore.SaveConfigBaseline.

Keys present in the baseline but absent from live are silently ignored (treat as "unknown", not "drift") — this respects the P4.1 graceful-skip contract for optional kernel features.

The baseline is NOT mutated on drift — the caller (P4.4) handles ack/update.

type Pattern 

type Pattern struct {
	Name       string
	Conditions []PatternCondition
	MinMatch   int     // how many conditions must fire
	Priority   int     // higher = checked first
	MinStr     float64 // minimum evidence strength to count (default 0.01)
	Narrative  string  // human-readable root cause sentence
}

Pattern is a named failure pattern that matches a combination of evidence signals.

func MatchPattern 

func MatchPattern(result *model.AnalysisResult) *Pattern

MatchPattern iterates the pattern library and returns the first matching pattern, or nil.

type PatternCondition 

type PatternCondition struct {
	EvidenceID  string
	MinStrength float64 // 0 = any non-zero strength
	Required    bool    // when true, this condition MUST match for the

}

PatternCondition is a single evidence requirement within a pattern.

type PgFaultEntry 

type PgFaultEntry struct {
	PID        int
	Comm       string
	AvgUs      float64
	MajorCount int
	TotalCount int
}

PgFaultEntry holds page fault latency for one process (watchdog probe).

type Player 

type Player struct {
	Engine *Engine
	// contains filtered or unexported fields
}

Player replays recorded frames through a virtual engine.

func NewPlayer 

func NewPlayer(r io.Reader, historySize int) (*Player, error)

NewPlayer creates a player from a recorded file (JSON lines).

func (*Player) Base 

func (p *Player) Base() *Engine

Base returns the underlying engine.

func (*Player) Index 

func (p *Player) Index() int

Index returns the next frame index.

func (*Player) Len 

func (p *Player) Len() int

Len returns the number of frames available.

func (*Player) Seek 

func (p *Player) Seek(i int) (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)

Seek jumps to a frame index and returns that frame.

func (*Player) Tick 

func (p *Player) Tick() (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)

Tick replays the next recorded frame (or the last frame if at EOF).

type ProbabilisticCausalGraph 

type ProbabilisticCausalGraph struct {
	// contains filtered or unexported fields
}

ProbabilisticCausalGraph maintains the learned causal network.

func NewProbabilisticCausalGraph 

func NewProbabilisticCausalGraph() *ProbabilisticCausalGraph

NewProbabilisticCausalGraph creates an empty causal graph.

func (*ProbabilisticCausalGraph) InferRootCauses 

func (g *ProbabilisticCausalGraph) InferRootCauses(fired map[string]float64, topN int) []CausalRootCause

InferRootCauses runs belief propagation and returns root causes ranked by posterior probability. A root cause is a node with high posterior but low incoming probability (i.e., it explains other symptoms but is not explained by other observed evidence).

func (*ProbabilisticCausalGraph) Observe 

func (g *ProbabilisticCausalGraph) Observe(fired map[string]float64)

Observe records a snapshot of fired evidence for learning.

type Probe 

type Probe struct {
	Name string // human-readable name, e.g. "top_cpu_processes"
	// Triggers is the set of evidence IDs that cause this probe to be
	// considered. The probe also requires Strength >= MinStrength.
	Triggers    []string
	MinStrength float64
	// Cmd + Args define the shell command. NO user-data interpolation here:
	// the args are static, by design.
	Cmd  string
	Args []string
}

Probe describes one investigation step.

type ProbeFindings 

type ProbeFindings struct {
	StartTime     time.Time
	Duration      time.Duration
	Pack          string // "auto", "offcpu", "iolatency", etc.
	Bottleneck    string // which RCA bottleneck this reinforces
	ConfBoost     int    // how much to boost RCA confidence
	Summary       string // one-line summary for overview
	OffCPUWaiters []OffCPUEntry
	IOLatency     []IOLatEntry
	LockWaiters   []LockEntry
	TCPRetrans    []TCPRetransEntry
	NetThroughput []NetThroughputEntry
	TCPRTT        []TCPRTTEntry
	TCPConnLat    []TCPConnLatEntry

	// Watchdog probe findings
	RunQLat        []RunQLatEntry
	WBStall        []WBStallEntry
	PgFault        []PgFaultEntry
	SwapEvict      []SwapEvictEntry
	SyscallDissect []SyscallDissectEntry
	SockIO         []SockIOEntry
}

ProbeFindings holds the output of a probe session.

type ProbeManager 

type ProbeManager struct {
	// contains filtered or unexported fields
}

ProbeManager manages the lifecycle of a probe session.

func NewProbeManager 

func NewProbeManager() *ProbeManager

NewProbeManager creates a new probe manager.

func (*ProbeManager) Findings 

func (pm *ProbeManager) Findings() *ProbeFindings

Findings returns the probe findings (nil if not done).

func (*ProbeManager) Pack 

func (pm *ProbeManager) Pack() string

Pack returns the current pack name.

func (*ProbeManager) ProbePack 

func (pm *ProbeManager) ProbePack() string

ProbePack implements probeQuerier.

func (*ProbeManager) ProbeSecsLeft 

func (pm *ProbeManager) ProbeSecsLeft() int

ProbeSecsLeft implements probeQuerier.

func (*ProbeManager) ProbeState 

func (pm *ProbeManager) ProbeState() int

ProbeState implements probeQuerier for the UI.

func (*ProbeManager) ProbeSummary 

func (pm *ProbeManager) ProbeSummary() string

ProbeSummary implements probeQuerier.

func (*ProbeManager) SecondsLeft 

func (pm *ProbeManager) SecondsLeft() int

SecondsLeft returns seconds remaining if running, 0 otherwise.

func (*ProbeManager) Start 

func (pm *ProbeManager) Start(pack string) error

Start initiates a probe session. Returns error if already running.

func (*ProbeManager) StartDomain 

func (pm *ProbeManager) StartDomain(domain string) error

StartDomain initiates a watchdog-triggered domain-specific probe session.

func (*ProbeManager) State 

func (pm *ProbeManager) State() ProbeState

State returns the current probe state.

func (*ProbeManager) Tick 

func (pm *ProbeManager) Tick()

Tick checks for state transitions. Called each UI tick (~1s). The Running→Done transition is handled by the goroutine. Done results persist until the next probe is started (no time-based expiry).

type ProbeRunner 

type ProbeRunner struct {
	// contains filtered or unexported fields
}

ProbeRunner is the per-engine probe scheduler.

func NewProbeRunner 

func NewProbeRunner() *ProbeRunner

NewProbeRunner constructs the runner. Honors XTOP_PROBES.

func (*ProbeRunner) Enabled 

func (r *ProbeRunner) Enabled() bool

Enabled reports whether probes will actually run.

func (*ProbeRunner) MaybeRun 

func (r *ProbeRunner) MaybeRun(result *model.AnalysisResult) []model.ProbeResult

MaybeRun considers the result and dispatches probes for any matching evidence. Probes run in goroutines but the runner waits up to a small budget for them to finish before returning so the result is attachable.

Returns the slice of probe results captured this tick (may be empty).

func (*ProbeRunner) SetEnabled 

func (r *ProbeRunner) SetEnabled(v bool)

SetEnabled forces enable/disable (for the CLI helper).

type ProbeState 

type ProbeState int

ProbeState represents the lifecycle of a probe session. 

const (
	ProbeIdle    ProbeState = 0
	ProbeRunning ProbeState = 1
	ProbeDone    ProbeState = 2
)

func (ProbeState) String 

func (s ProbeState) String() string

type ProcessEntry 

type ProcessEntry struct {
	PID    int
	Comm   string
	CPUPct float64
	IOMBs  float64 // read + write MB/s
	RSS    uint64
	State  string
}

ProcessEntry is a lightweight record of one process at one point in time.

type ProcessHistory 

type ProcessHistory struct {
	// contains filtered or unexported fields
}

ProcessHistory maintains a ring buffer of process snapshots for temporal culprit analysis. Stores the last N ticks worth of top process data (~5 min at 3s interval).

func NewProcessHistory 

func NewProcessHistory(capacity int) *ProcessHistory

NewProcessHistory creates a process history buffer.

func (*ProcessHistory) FindCPUCulprit 

func (ph *ProcessHistory) FindCPUCulprit() (comm string, pid int, appearances int)

FindCPUCulprit looks back through process history to find the process that has been consistently in the top CPU consumers. This finds the ROOT CAUSE (the process that started consuming CPU first), not just whoever is on top at the current moment.

func (*ProcessHistory) FindIOCulprit 

func (ph *ProcessHistory) FindIOCulprit() (comm string, pid int, appearances int)

FindIOCulprit looks back to find the process consistently generating the most IO.

func (*ProcessHistory) FindMemCulprit 

func (ph *ProcessHistory) FindMemCulprit() (comm string, pid int, appearances int)

FindMemCulprit looks back to find the process consistently using the most memory.

func (*ProcessHistory) Record 

func (ph *ProcessHistory) Record(rates *model.RateSnapshot)

Record stores the top processes from the current tick.

type ProcessSnapshot 

type ProcessSnapshot struct {
	Timestamp time.Time
	TopCPU    []ProcessEntry // top 10 by CPU%
	TopIO     []ProcessEntry // top 10 by IO MB/s
	TopMem    []ProcessEntry // top 10 by RSS
}

ProcessSnapshot records the top CPU/IO/Memory consumers at one point in time.

type RCAIncident 

type RCAIncident struct {
	StartedAt   time.Time `json:"started_at"`
	EndedAt     time.Time `json:"ended_at,omitempty"`
	DurationSec int       `json:"duration_sec"`
	Bottleneck  string    `json:"bottleneck"`
	Pattern     string    `json:"pattern,omitempty"`
	PeakScore   int       `json:"peak_score"`
	Confidence  int       `json:"confidence"`
	Culprit     string    `json:"culprit,omitempty"`
	CulpritApp  string    `json:"culprit_app,omitempty"`
	RootCause   string    `json:"root_cause,omitempty"`
	Evidence    []string  `json:"evidence,omitempty"`

	// EvidenceIDs are the firing evidence IDs at incident peak — used by
	// DiffAgainstHistory to compute which signals are new or missing this
	// time around. Older history files may lack this field; diff gracefully
	// degrades to using the narrative Evidence strings instead.
	EvidenceIDs []string `json:"evidence_ids,omitempty"`

	// Signature for similarity matching — stable hash of firing evidence IDs
	Signature string `json:"signature"`

	// Resolution — set if we detected how the incident ended
	Resolution string `json:"resolution,omitempty"` // "auto-recovered", "sustained", "escalated"

	// Lifecycle (Phase 1: verdict discipline).
	// State is the current lifecycle stage; ConfirmedAt is the wall-clock at
	// which Suspected→Confirmed transition occurred. Both empty/zero on older
	// history records loaded from disk; readers must tolerate that.
	State       IncidentState `json:"state,omitempty"`
	ConfirmedAt time.Time     `json:"confirmed_at,omitempty"`

	// Phase 7: changes that landed in the 30 minutes preceding the
	// Confirmed transition. Captured at promotion time to support "what
	// changed?" forensics; immutable after that.
	ChangesAtConfirm []model.SystemChange `json:"changes_at_confirm,omitempty"`
	// Phase 7: human-readable summary of fleet peers reporting similar
	// evidence at the same time.
	FleetPeersAtConfirm string `json:"fleet_peers_at_confirm,omitempty"`
}

RCAIncident is a recorded past incident for learning.

type Recorder 

type Recorder struct {
	Engine *Engine
	// contains filtered or unexported fields
}

Recorder wraps an engine and records every tick to a file.

func NewRecorder 

func NewRecorder(eng *Engine, w io.Writer) *Recorder

NewRecorder creates a recorder that writes JSON lines to w.

func (*Recorder) Base 

func (r *Recorder) Base() *Engine

Base returns the underlying engine.

func (*Recorder) Close 

func (r *Recorder) Close()

Close flushes and closes the recorder.

func (*Recorder) RecordTick 

func (r *Recorder) RecordTick() (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)

RecordTick calls the engine's Tick and records the result.

func (*Recorder) RecordTickWithProbe 

func (r *Recorder) RecordTickWithProbe(probe *ProbeFindings) (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)

RecordTickWithProbe calls the engine's Tick and records the result along with probe findings.

func (*Recorder) Tick 

func (r *Recorder) Tick() (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)

Tick records a frame and returns it.

type ResourceGuard 

type ResourceGuard struct {
	// contains filtered or unexported fields
}

ResourceGuard throttles xtop's own resource usage when the host is busy. The goal: never make a bad day worse by having the observability tool compete with the workload it's supposed to be observing.

Three signals drive the decision:

  1. Host load ratio — LoadAvg1 / NumCPUs. > 1.5 is "stressed", > 3.0 is "thrashing". These thresholds are per-core so a 32-core box doesn't trip just because its load is 12.
  2. Host CPU-busy% — rolled-up from rates. Useful when LoadAvg1 lies (e.g. many D-state procs pile up load without using CPU).
  3. xtop's own CPU% — parsed from /proc/self/stat between ticks. When WE are the cause of the pain, we self-throttle more aggressively.

Four levels of response: 

Level 0 (normal):      everything runs at full cadence
Level 1 (caution):     skip app deep-metric probes + log tailer
Level 2 (degraded):    also skip trace correlator + eBPF watchdog triggers
Level 3 (minimal):     also skip deep scan; interval up to 2× base

Hysteresis: we escalate after 3 consecutive ticks above threshold, and de-escalate only after 10 consecutive ticks below. Prevents flapping when load is noisy.

Self-attribution: when our own CPU climbs above ownCPUBudgetPct AND the host is loaded, we're a contributor — force the level higher than the host signals alone would suggest. This closes the "xtop made it worse" feedback loop that an early version of this code shipped with.

Opt-in: must be enabled via XTOP_GUARD=1 (or explicit Enable()). Default off so existing deployments see no behavior change.

func NewResourceGuard 

func NewResourceGuard(numCPUs, baseIntervalSec int) *ResourceGuard

NewResourceGuard constructs a guard with sensible defaults, then overlays any environment variables. Returns a disabled guard when XTOP_GUARD is anything other than "1" — the caller can still use Advice(), but the returned level will always be 0.

func (*ResourceGuard) Advise 

func (g *ResourceGuard) Advise(loadAvg1, hostBusyPct float64) GuardAdvice

Advise is called once per tick. It reads the host signals plus xtop's own CPU consumption and returns what to skip this cycle. The returned IntervalSec is authoritative — callers should sleep that many seconds until the next Tick.

func (*ResourceGuard) AllowDeepAppAnalysis 

func (g *ResourceGuard) AllowDeepAppAnalysis(lastRun time.Time) bool

AllowDeepAppAnalysis returns true when the system is calm enough to afford expensive per-app deep diagnostics (subprocess spawns, HTTP scrapes, etc.). This is stricter than SkipAppDeep: even at guard level 0 we throttle deep analysis to once every 30s to avoid persistent background load.

func (*ResourceGuard) CurrentIntervalSec 

func (g *ResourceGuard) CurrentIntervalSec() int

CurrentIntervalSec exposes the live interval for daemons/tickers that need to sleep between ticks. Readers atomic so no lock needed.

func (*ResourceGuard) Enabled 

func (g *ResourceGuard) Enabled() bool

Enabled reports whether the guard is active. When false, Advise returns a level-0 advice with no skips so callers can uniformly consult it.

func (*ResourceGuard) SetEnabled 

func (g *ResourceGuard) SetEnabled(on bool)

SetEnabled toggles the guard at runtime. Used by the UI's 'g' key binding so operators can disable the safety throttle from inside a running session without having to exit, re-export XTOP_GUARD=0, and restart. When disabled, the level is forced back to 0 so the engine immediately re-runs the deep probes that were being skipped.

type RingBuf 

type RingBuf[T any] struct {
	// contains filtered or unexported fields
}

RingBuf is a generic ring buffer with thread-safe access.

func NewRingBuf 

func NewRingBuf[T any](capacity int) *RingBuf[T]

NewRingBuf creates a new ring buffer with the given capacity.

func (*RingBuf[T]) Get 

func (r *RingBuf[T]) Get(i int) (T, bool)

Get returns the i-th element (0 = oldest).

func (*RingBuf[T]) Latest 

func (r *RingBuf[T]) Latest() (T, bool)

Latest returns the most recently pushed element.

func (*RingBuf[T]) Len 

func (r *RingBuf[T]) Len() int

Len returns the number of elements in the buffer.

func (*RingBuf[T]) Push 

func (r *RingBuf[T]) Push(v T)

Push adds an element to the ring buffer.

func (*RingBuf[T]) Slice 

func (r *RingBuf[T]) Slice() []T

Slice returns all elements oldest-first.

type RunQLatEntry 

type RunQLatEntry struct {
	PID   int
	Comm  string
	AvgUs float64
	MaxUs float64
	Count int
}

RunQLatEntry holds run queue latency for one process (watchdog probe).

type Runbook 

type Runbook struct {
	Path    string         `json:"path"`
	Name    string         `json:"name"`
	Match   RunbookMatcher `json:"match"`
	Content string         `json:"content"` // markdown body without frontmatter
	// UpdatedAt is the file's mtime — the UI can show "runbook last edited Nd ago".
	UpdatedAt time.Time `json:"updated_at"`
}

Runbook is a single loaded runbook document. The parsed matcher fields live on Match; the rendered markdown body lives on Content.

type RunbookLibrary 

type RunbookLibrary struct {
	// contains filtered or unexported fields
}

RunbookLibrary loads operator-authored runbooks from ~/.xtop/runbooks/*.md, matches them against incident context, and attaches the best match to the current RCA result.

Format — a markdown file with a small YAML-ish frontmatter block: 

---
name: Nginx worker saturation
bottleneck: cpu, network
app: nginx
culprit: nginx
evidence: runqlat_high, conn_queue_overflow
signature: cpu|runqlat_high,
---

## Diagnosis
Workers are maxed out — check:
```bash
nginx -T | grep worker_processes
```

## Fix
Set `worker_processes auto;` and reload.

Matching is additive: each frontmatter field adds to a score only when the runbook's value overlaps with the incident's. A runbook with no match fields acts as a generic fallback (score 0 unless it specifies otherwise).

func NewRunbookLibrary 

func NewRunbookLibrary() *RunbookLibrary

NewRunbookLibrary creates a library pointing at ~/.xtop/runbooks/.

func (*RunbookLibrary) All 

func (l *RunbookLibrary) All() []Runbook

All returns a copy of the loaded runbooks (for listing in a picker).

func (*RunbookLibrary) Lookup 

func (l *RunbookLibrary) Lookup(path string) *Runbook

Lookup returns the full Runbook for a matched path. Used by the UI to show the rendered content inside a drawer / detail panel.

func (*RunbookLibrary) Match 

func (l *RunbookLibrary) Match(result *model.AnalysisResult) *model.RunbookMatch

Match returns the best-scoring runbook for the given result, or nil if no runbook matches (score <= 0 or below MinScore).

func (*RunbookLibrary) Reload 

func (l *RunbookLibrary) Reload() error

Reload walks the runbook directory and reparses every *.md file. Called on creation and on-demand by Match() once per reloadEvery interval, so edits pick up without a daemon restart.

type RunbookMatcher 

type RunbookMatcher struct {
	Bottleneck      []string `json:"bottleneck,omitempty"`
	AppContains     []string `json:"app_contains,omitempty"`
	CulpritContains []string `json:"culprit_contains,omitempty"`
	EvidenceAny     []string `json:"evidence_any,omitempty"`
	Signature       []string `json:"signature,omitempty"`
	// MinScore lets a runbook say "only fire if my score hits N" — otherwise
	// any nonzero match wins the best-of-library competition.
	MinScore int `json:"min_score,omitempty"`
}

RunbookMatcher lists the criteria (all lowercased) for choosing this runbook. An empty slice on a field means "don't gate on this field." All scoring is substring-based for apps/culprits and exact for bottlenecks/evidence IDs.

type SLOConfig 

type SLOConfig struct {
	Policies []SLOPolicyConfig `json:"policies,omitempty"`
}

SLOConfig holds SLO configuration.

type SLOPolicy 

type SLOPolicy struct {
	Name      string  // e.g. "p95<200ms"
	Metric    string  // "p95", "p99", "cpu", "mem", "io_psi", "cpu_psi"
	Operator  string  // "<", ">", "<=", ">="
	Threshold float64 // threshold value
	Unit      string  // "ms", "%", ""
}

SLOPolicy defines a service level objective.

func ParseSLOFlag 

func ParseSLOFlag(s string) (SLOPolicy, error)

ParseSLOFlag parses a flag string like "p95<200ms" or "cpu<80%" into an SLOPolicy.

type SLOPolicyConfig 

type SLOPolicyConfig struct {
	Name string `json:"name"` // e.g. "cpu<80%", "io_psi<5%"
}

SLOPolicyConfig is a serializable SLO policy.

type SLOResult 

type SLOResult struct {
	Policy    SLOPolicy
	Current   float64
	Passed    bool
	Message   string
	CheckedAt time.Time
}

SLOResult holds the evaluation of one SLO policy.

func EvaluateSLOs 

func EvaluateSLOs(policies []SLOPolicy, snap *model.Snapshot, rates *model.RateSnapshot, result *model.AnalysisResult) []SLOResult

EvaluateSLOs evaluates a list of SLO policies against current metrics.

type SeasonalTracker 

type SeasonalTracker struct {
	// contains filtered or unexported fields
}

SeasonalTracker maintains per-hour-of-day baselines for key metrics. This allows the engine to learn that "CPU is always high at 2AM during backups".

func NewSeasonalTracker 

func NewSeasonalTracker(alpha float64) *SeasonalTracker

NewSeasonalTracker creates a tracker with per-hour EWMA baselines.

func (*SeasonalTracker) IsSuppressed 

func (st *SeasonalTracker) IsSuppressed(id string, value float64, hour int, nSigma float64) bool

IsSuppressed returns true if the value is within normal range for this hour. Used to suppress alerts for known recurring patterns (e.g. nightly backups).

func (*SeasonalTracker) SeasonalBaseline 

func (st *SeasonalTracker) SeasonalBaseline(id string, hour int) (mean, std float64, ok bool)

SeasonalBaseline returns the baseline mean+std for a specific hour.

func (*SeasonalTracker) Update 

func (st *SeasonalTracker) Update(id string, value float64, hour int)

Update feeds a value for the current hour of day.

type SockIOEntry 

type SockIOEntry struct {
	PID       int
	Comm      string
	DstAddr   string // "10.0.0.5:5432"
	Service   string // "postgres"
	TxBytes   uint64
	RxBytes   uint64
	AvgWaitMs float64
	MaxWaitMs float64
	RecvCount int
}

SockIOEntry holds per-PID per-connection TCP IO data (watchdog probe).

type SwapEvictEntry 

type SwapEvictEntry struct {
	PID        int
	Comm       string
	ReadPages  uint64
	WritePages uint64
}

SwapEvictEntry holds swap activity for one process (watchdog probe).

type SyscallDissectEntry 

type SyscallDissectEntry struct {
	PID       int
	Comm      string
	Breakdown []SyscallGroupTime // sorted by TotalPct desc
	TotalNs   uint64
}

SyscallDissectEntry holds per-PID syscall time breakdown (watchdog probe).

type SyscallGroupTime 

type SyscallGroupTime struct {
	Group    string // "read", "write", "lock/sync", "poll", "sleep", "mmap", "open/close", "other"
	TotalNs  uint64
	Count    uint32
	MaxNs    uint32
	TotalPct float64 // % of this PID's total syscall time
}

SyscallGroupTime holds time spent in one syscall group for a PID.

type TCPConnLatEntry 

type TCPConnLatEntry struct {
	PID     int
	Comm    string
	DstAddr string
	AvgMs   float64
	MaxMs   float64
	Count   int
}

TCPConnLatEntry holds TCP connection establishment latency.

type TCPRTTEntry 

type TCPRTTEntry struct {
	DstAddr  string
	AvgRTTMs float64
	MinRTTMs float64
	MaxRTTMs float64
	Samples  int
	TopComm  string
}

TCPRTTEntry holds RTT data for one remote endpoint.

type TCPRetransEntry 

type TCPRetransEntry struct {
	PID     int
	Comm    string
	Retrans int // per second
	Iface   string
}

TCPRetransEntry holds TCP retransmit attribution for one process.

type ThresholdOverride 

type ThresholdOverride struct {
	Warn float64
	Crit float64
}

ThresholdOverride holds custom warn/crit thresholds for an evidence ID.

type ThresholdProfile 

type ThresholdProfile map[string]ThresholdOverride

ThresholdProfile maps evidence IDs to threshold overrides. 

var ActiveProfile ThresholdProfile

ActiveProfile is the currently loaded threshold profile. Set at startup from config; nil means use hardcoded defaults.

type Ticker 

type Ticker interface {
	Tick() (*model.Snapshot, *model.RateSnapshot, *model.AnalysisResult)
	Base() *Engine
}

Ticker abstracts a data source that can produce snapshots.

func NewInstrumentedTicker 

func NewInstrumentedTicker(inner Ticker, store *MetricsStore) Ticker

NewInstrumentedTicker wraps a ticker and updates the metrics store.

type TimeSeries1h 

type TimeSeries1h struct {
	Timestamp time.Time
	CPUBusy   float64
	MemPct    float64
	IOAwait   float64
	NetDrops  float64
	Health    int
	Count     int // samples in this hour
}

TimeSeries1h holds a 1-hour aggregated data point.

type TimeSeries1m 

type TimeSeries1m struct {
	Timestamp time.Time
	CPUBusy   float64
	MemPct    float64
	IOAwait   float64
	NetDrops  float64
	Health    int
	Count     int // samples in this minute
}

TimeSeries1m holds a 1-minute aggregated data point.

type TimeSeries1s 

type TimeSeries1s struct {
	Timestamp time.Time
	Health    int
	Score     int
	CPUBusy   float64
	MemPct    float64
	IOPSI     float64
	TopPID    int
	TopComm   string
	IOAwait   float64
	NetDrops  float64
}

TimeSeries1s holds a single high-resolution (per-sample) data point.

type TimeSeries10s 

type TimeSeries10s struct {
	Timestamp time.Time
	HealthMax int // worst health in window
	ScoreMax  int
	CPUMin    float64
	CPUMax    float64
	CPUAvg    float64
	MemMin    float64
	MemMax    float64
	MemAvg    float64
	IOPSIMax  float64
}

TimeSeries10s is a 10-second resolution aggregate sample (used by RingBuf-based pipeline).

type TopologyCorrelator 

type TopologyCorrelator struct {
	// contains filtered or unexported fields
}

TopologyCorrelator maintains the infrastructure graph and correlates incidents.

func NewTopologyCorrelator 

func NewTopologyCorrelator() *TopologyCorrelator

NewTopologyCorrelator creates a new topology-aware correlator.

func (*TopologyCorrelator) Correlate 

func (tc *TopologyCorrelator) Correlate(hostID string, window time.Duration) []IncidentCorrelation

Correlate finds all hosts whose current incidents are likely caused by or related to the given host's incident. Returns correlations ranked by likelihood.

func (*TopologyCorrelator) RecordIncident 

func (tc *TopologyCorrelator) RecordIncident(hostID string, incident *model.HostIncident)

RecordIncident stores an incident for correlation.

func (*TopologyCorrelator) RegisterHost 

func (tc *TopologyCorrelator) RegisterHost(host *HostTopology)

RegisterHost adds or updates a host in the topology graph.

type TopologyRole 

type TopologyRole string

TopologyRole describes a host's position in the infrastructure graph. 

const (
	RoleLoadBalancer TopologyRole = "lb"
	RoleWebServer    TopologyRole = "web"
	RoleAppServer    TopologyRole = "app"
	RoleDatabase     TopologyRole = "db"
	RoleCache        TopologyRole = "cache"
	RoleQueue        TopologyRole = "queue"
	RoleUnknown      TopologyRole = "unknown"
)

type TraceArmer 

type TraceArmer struct {
	// contains filtered or unexported fields
}

TraceArmer captures full A→Z reasoning of a single tick into one JSON + one markdown file in ~/.xtop/traces/. It is the user-facing verification tool: every claim in the verdict has a line of evidence in the trace, and the markdown reads like a sysadmin's notebook.

This is NOT a continuous trace stream — that would be unmanageable in production at 20 verdicts/min/host. It is arm-once or transition-only.

Activation: env at process start (XTOP_TRACE_NEXT=1 / XTOP_TRACE_ON_CONFIRMED=1) or programmatically via Arm(). Each TraceModeNext dump self-disarms.

func NewTraceArmer 

func NewTraceArmer(dir string) *TraceArmer

NewTraceArmer reads XTOP_TRACE_* env to decide initial mode. dir defaults to ~/.xtop/traces if empty.

func (*TraceArmer) Arm 

func (t *TraceArmer) Arm(m TraceMode)

Arm sets the mode programmatically. Used by `xtop trace --once`.

func (*TraceArmer) MaybeDump 

func (t *TraceArmer) MaybeDump(
	snap *model.Snapshot,
	rates *model.RateSnapshot,
	result *model.AnalysisResult,
	hist *History,
	active *RCAIncident,
)

MaybeDump is called every Tick. Decides whether to write a trace based on mode + lifecycle state, and writes both the .json and .md files when so.

func (*TraceArmer) Mode 

func (t *TraceArmer) Mode() TraceMode

Mode reports the current mode.

type TraceCorrelator 

type TraceCorrelator struct {
	// contains filtered or unexported fields
}

TraceCorrelator tails a simple JSONL feed of OpenTelemetry trace summaries and returns samples that overlap the current incident window.

Why a file feed instead of OTLP: shipping an OTLP receiver inside xtop would pull a heavy dependency tree (otel-collector, grpc, protobuf) that most operators don't need. A single JSONL file any collector pipeline can produce is sustainably portable — we stay a tiny binary and operators pick whichever OTel backend they already run.

Feed file (default ~/.xtop/otel-samples.jsonl, overridable via $XTOP_OTEL_SAMPLES_FILE): one TraceSample JSON per line, appended by the operator's OTel collector. Lines older than 2 h are ignored. A sensible minimal recipe — using the otel-collector `file` exporter — is documented in packaging/otel/README.md.

func NewTraceCorrelator 

func NewTraceCorrelator() *TraceCorrelator

NewTraceCorrelator returns a correlator with defaults. If the feed file doesn't exist we simply hold an empty sample set and stay cheap; no error, because the feed is fundamentally optional.

func (*TraceCorrelator) Flush 

func (c *TraceCorrelator) Flush()

Flush drops the in-memory ring of cached samples + resets the read offset so the next refresh re-tails from current EOF. Called by the Guardian under memory pressure. Cheap.

func (*TraceCorrelator) Observe 

func (c *TraceCorrelator) Observe(result *model.AnalysisResult) []model.TraceSample

Observe returns trace samples that overlap the incident's current window. Returns nil when the feed file is absent / unreadable — silent because OTel correlation is an optional feature.

type TraceFile 

type TraceFile struct {
	Schema    string    `json:"schema"`
	WrittenAt time.Time `json:"written_at"`
	Hostname  string    `json:"hostname,omitempty"`

	// Inputs the engine actually consulted this tick.
	Inputs traceInputs `json:"inputs"`

	// Per-domain analysis.
	Domains []traceDomain `json:"domains"`

	// Cross-domain analysis.
	Correlations  []model.MetricCorrelation  `json:"correlations,omitempty"`
	Blame         []model.BlameEntry         `json:"blame,omitempty"`
	CausalChain   string                     `json:"causal_chain,omitempty"`
	AppAnomalies  []model.AppBehaviorAnomaly `json:"app_anomalies,omitempty"`
	DriftWarnings []model.DegradationWarning `json:"drift_warnings,omitempty"`
	Probes        []model.ProbeResult        `json:"probes,omitempty"`
	Changes       []model.SystemChange       `json:"changes,omitempty"`
	FleetPeers    string                     `json:"fleet_peers,omitempty"` // human-readable peer correlation summary

	// Gate audit — why the verdict landed where it did.
	GateAudit traceGateAudit `json:"gate_audit"`

	// Lifecycle.
	Incident *RCAIncident `json:"incident,omitempty"`

	// Final verdict.
	Verdict traceVerdict `json:"verdict"`
}

TraceFile is the on-disk JSON shape. Stable enough for tooling to consume, but not yet a public API; field names may shift.

type TraceMode 

type TraceMode int

TraceMode controls when a TraceArmer dumps a trace. 

const (
	TraceModeOff         TraceMode = 0
	TraceModeNext        TraceMode = 1 // dump the next tick, then disarm
	TraceModeOnConfirmed TraceMode = 2 // dump on every Suspected→Confirmed transition
)

type UsageRecorder 

type UsageRecorder struct {
	// contains filtered or unexported fields
}

UsageRecorder appends a compact per-minute utilization rollup to ~/.xtop/usage-history.jsonl. Over time this becomes the data source for right-sizing recommendations (see `xtop cost`).

Each rollup line is ~150 bytes, so a minute's data for 90 days lands at ~20 MB even in the worst case — modest but still too large to keep in memory, hence the streaming file format.

Lock ordering (must be respected to avoid deadlock): 

r.mu → (release) → r.fileMu

r.mu guards in-memory state (samples, currentMin, lastPruneAt). r.fileMu guards ALL disk IO on r.path (both the append in writeFlush and the read-modify-write in prune). The two mutexes are NEVER held at the same time: prepareFlushLocked runs under r.mu and produces serialised bytes; writeFlush takes only fileMu after r.mu has been released. prune also snapshots config under r.mu, releases r.mu, then takes fileMu.

func NewUsageRecorder 

func NewUsageRecorder() *UsageRecorder

NewUsageRecorder opens (or creates) the rollup file under ~/.xtop/.

func (*UsageRecorder) Flush 

func (r *UsageRecorder) Flush(numCPUs int, memTotal uint64)

Flush forces the current in-memory window to disk. Called on shutdown. r.mu is released before file IO.

func (*UsageRecorder) Observe 

func (r *UsageRecorder) Observe(cpuPct, memPct, ioPct, load1 float64, numCPUs int, memTotal uint64)

Observe takes one tick's aggregate numbers. Cheap: samples are accumulated in memory until the minute rolls over, at which point a single line is flushed to disk.

r.mu is released before any file IO so the Tick goroutine never blocks on prune's disk read-modify-write.

type UsageRollup 

type UsageRollup struct {
	Minute    time.Time `json:"minute"`
	Samples   int       `json:"samples"`
	CPU       UsageStat `json:"cpu"`
	Mem       UsageStat `json:"mem"`
	IO        UsageStat `json:"io"`
	LoadRatio UsageStat `json:"load_ratio"`
	NumCPUs   int       `json:"num_cpus,omitempty"`
	MemTotal  uint64    `json:"mem_total_bytes,omitempty"`
}

UsageRollup is one minute of aggregated usage, persisted as a JSON line. Fields are tagged so `xtop cost` and hub ingestion can round-trip them.

type UsageStat 

type UsageStat struct {
	Max float64 `json:"max"`
	P95 float64 `json:"p95"`
	P50 float64 `json:"p50"`
	Avg float64 `json:"avg"`
}

UsageStat holds summary statistics for one metric across a minute.

type WBStallEntry 

type WBStallEntry struct {
	PID        int
	Comm       string
	Count      int
	TotalPages int64
}

WBStallEntry holds writeback stall data for one process (watchdog probe).

type WatchdogTrigger 

type WatchdogTrigger struct {
	// contains filtered or unexported fields
}

WatchdogTrigger monitors RCA results and auto-triggers domain probes when a bottleneck is detected with sufficient confidence.

func NewWatchdogTrigger 

func NewWatchdogTrigger() *WatchdogTrigger

NewWatchdogTrigger creates a new watchdog trigger with defaults.

func (*WatchdogTrigger) Check 

func (w *WatchdogTrigger) Check(result *model.AnalysisResult) string

Check returns the domain name if the watchdog should trigger, "" otherwise.

type WorkloadType 

type WorkloadType string

WorkloadType identifies the kind of workload running on the host. 

const (
	WorkloadWeb          WorkloadType = "web"
	WorkloadDatabase     WorkloadType = "database"
	WorkloadCache        WorkloadType = "cache"
	WorkloadBatch        WorkloadType = "batch"
	WorkloadMessageQueue WorkloadType = "message_queue"
	WorkloadKubernetes   WorkloadType = "kubernetes"
	WorkloadUnknown      WorkloadType = "unknown"
)

func DetectWorkloadType 

func DetectWorkloadType(curr *model.Snapshot) WorkloadType

DetectWorkloadType infers the workload type from running processes and apps.

type ZScoreTracker 

type ZScoreTracker struct {
	// contains filtered or unexported fields
}

ZScoreTracker maintains sliding windows for multiple metrics.

func NewZScoreTracker 

func NewZScoreTracker(windowSize int) *ZScoreTracker

NewZScoreTracker creates a tracker with the given window size (number of samples).

func (*ZScoreTracker) MeanStd 

func (zt *ZScoreTracker) MeanStd(id string) (float64, float64, bool)

MeanStd returns the window mean and standard deviation.

func (*ZScoreTracker) Push 

func (zt *ZScoreTracker) Push(id string, value float64)

Push adds a value for a metric to its sliding window.

func (*ZScoreTracker) ZScore 

func (zt *ZScoreTracker) ZScore(id string, value float64) float64

ZScore returns the z-score for a value against the sliding window. Returns 0 if insufficient data.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.