xtop

command module

v0.50.0 Latest Latest Go to latest Published: Jul 13, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/ftahirops/xtop

Links

Open Source Insights

README ¶

xtop

Stop monitoring. Start diagnosing.

The next-generation Linux performance console that doesn't just show what is happening —
it tells you why it's happening, who is causing it, and how to fix it.

xtop detects the bottleneck, identifies the culprit process, and tells you exactly what to do.

Install Now • Why xtop? • Features • Full Install Guide • Docs

What is new in v0.50.0

The correctness release: a full adversarially-verified audit of the RCA engine (27 findings, 7 structural classes — all fixed) plus a 22-page / 486-parameter verification of every displayed value against live /proc, ss, df, and cgroup ground truth.

RCA engine correctness

No more misattribution: "hypervisor stealing CPU" can no longer fire without real steal; "swap thrashing" can no longer fire on non-swapping hosts (the defining evidence of every specific-cause pattern is now required)
Run-queue saturation measured from instantaneous procs_running, not the Load1 EWMA
Confidence is capped by evidence strength — weak signals can't display 93%
WHO always agrees with WHY (kernel slab leaks blame the kernel; hypervisor narratives blame the hypervisor)
One consistent frame: pinned RCA results render with their own snapshot, never mixed with live numbers
Evidence pipeline ordering fixed: Proxmox VM memory, journal Tier-1, and config-drift evidence now flow into scoring/verification before the verdict is finalized
Unverified verdicts are labeled (unverified); lean/fleet mode surfaces its reduced signal coverage

Reporting accuracy (verification sweep)

LVM/RAID hosts no longer double-count disk throughput/IOPS/top-writer (dm layers re-report the physical device's IO)
MySQL 8's event scheduler no longer trips a permanent bogus "kill this query" CRIT
Runtime census scans all of /proc (9 node processes now show as 9, not 1); apps RESOURCE SHARE memory uses full-tree RSS
Vhost-only web servers (nginx return 444) probe as WARN "port open", not down
Partial disk scans say so ("sizes are at-least"); warm-up shows "—" instead of misleading zeros
Thresholds page now documents the engine's real base thresholds and slot-based scoring

New

DiskGuard "TOP DIRECTORIES": du-style disjoint directory rollups in the existing scan pass
Apps table: Runtime column (docker/k8s/containerd/podman/native), self-explanatory Ranks column

What is new in v0.49.0

A large hardening + features release: every Critical/High audit finding fixed, two new RCA evidence subsystems, and a real CI pipeline. ~96 commits, test files nearly doubled (55 → 111), -race-clean, and the lean fleet agent stays free of TUI/SQLite (now CI-enforced).

New features

Journal-driven service RCA. systemd journal output is parsed structurally (journalctl -o json, not keyword grep) and classified into 7 typed failure signatures (crash/restart loop, OOM-kill, segfault/panic, resource exhaustion, dependency failure, config/auth error, error-rate spike). Findings become verifier-gated FactKindLogEvidence facts attached to the owning service and surface in that service's RCA narrative. Three-tier scope via --journal-rca=critical|all|off (default critical).
Config-drift RCA. Baselines RCA-relevant kernel/OS config (vm.swappiness, overcommit, somaxconn, nf_conntrack_max, THP, CPU governor, …), persists it, detects drift, and correlates drift timing+domain with anomaly onset to blame the right change. Detect + explain only — never auto-remediates; the narrative suggests a fix as text, nothing is ever written to /proc/sys. Toggle with --config-drift=on|off (default on).

Security hardening (fleet hub)

Auth enforced by default (refuses to start tokenless unless --allow-no-auth), constant-time token compare, request-body size cap (OOM-DoS), per-agent rate limiting with TTL pruning, client-facing error sanitization, agent→hub TLS verification on by default (--insecure opt-out), credential scrubbing from shipped process command-lines, hostname path-segment validation. MySQL probe password moved off the command line to MYSQL_PWD.

Stability & correctness

Outage-class bugs fixed: SMART disk-health parser buffer corruption (disk-failure prediction was unreliable); /health map data race; AdaptiveThresholdDB goroutine/ticker leak on shutdown; unbounded caches bounded (rdns, container-id, log-history, kubepods, signalOnsets, per-PID maps). Three further pre-existing data races surfaced by -race and fixed.
Collector correctness: allocstall summed across all zones (memory-stall signal was silently 0 on kernels ≥5.14); cgroup tree walk recurses to k8s depth (pod/container cgroups were invisible); tailFile seeks from end instead of reading multi-GB logs; IPv6 CLOSE_WAIT remote-IP attribution; Redis RESP blank-line panic guards; identity DB-probe timeouts; container-aware security scan (no longer skips low-PID processes inside containers).
RCA engine: per-metric Holt-Winters α/β (cross-metric contamination fixed); PrimaryScore now respects the trust gate (bonus stacking can no longer read critical/degraded while the verdict is Inconclusive); forecast/sigma narrative consistency.

TUI / web / tooling

TUI: NO_COLOR support, responsive widths, per-page scroll memory, loading spinner, ANSI-safe wrap. Web dashboard: responsive layout, render-on-change (no flicker), drawer fetch-abort, stale-stream banner, WCAG-AA contrast, keyboard nav.
CI added (.github/workflows/ci.yml): build, go vet, tests, -race, coverage, agent-leanness gate, Dependabot — where there was none.

All v0.48.0 features (NEXTGEN proof-system, multi-gate verifier, entity graph, replay corpus) remain intact and unchanged.

What is new in v0.48.0

NEXTGEN architecture milestone — five phases of the proof-system refactor land in production. The RCA engine now has typed evidence, an entity graph, a 5-gate verifier with strict abstention, and an offline replay corpus. The structural primitives required for a 0.1% false-positive precision target are all in place; what remains is operator-driven calibration via the captured corpus. See docs/NEXTGEN_STATUS.md for the honest per-phase landing report.

Phase 1 — Correctness & Stability. Removed 2 of 3 package-global RCA state pieces. Single finalization point (Engine.finalize) — score-band Health decision in one place, hysteresis as a separate late-stage call. Two engines in one process verified isolated by test.
Phase 2 — Typed Fact evidence. New model.Fact with 18 fields + JSON tags + provenance. Every signal becomes a Fact with per-metric Confidence (1.0 kernel-direct → 0.6 interpretation). 23 typed facts emitted per tick at full coverage (CPU 6, Memory 7, IO 6, Network 4). Lossless JSON roundtrip — facts are replay-ready.
Phase 3 — Entity graph. New model.Entity + model.EntityGraph with O(1) lookup, cycle-guarded ancestor walk. BuildEntityGraph() constructs process + cgroup ownership tree each tick (~100µs on 500-proc host). Fact.EntityID resolves through the graph for ownership reasoning.
Phase 4 — Multi-gate verifier with 4-tier output. model.VerifiedCause carrying Tier A confirmed / B verified / C probable / D inconclusive with full per-gate audit trail. 5 gates: signal_quality, ownership_consistency, temporal_ordering, baseline_deviation, counter_evidence. Abstain-by-default semantics — engine refuses to commit when proof is weak. Tier A reachable only when all 5 gates pass.
Phase 5 — Replay corpus. Automatic capture of non-OK ticks to ~/.xtop/incidents/. New xtop replay subcommand replays captured frames against the current verifier and reports per-mechanism tier-match (determinism check). When operators add labels (TP/FP/FN/TN), the harness computes per-mechanism precision — the substrate of the NEXTGEN §7 precision program.

Honest about the 0.1% target. The architecture is achievable; the number isn't yet measured. Operators must label the captured corpus to convert "the engine abstains a lot" into "FP rate is 0.X% per mechanism on a 500-incident labeled corpus." Anyone claiming the number without showing the labeled corpus is selling fiction.

18 structural invariants now run on every analysis tick — protecting RCA shape (I1-9), fact well-formedness (I10-11), entity graph integrity (I12-14), verifier output contract (I15-17), and replay-harness contract (I18).

All v0.47.x features (PHP-FPM diagnostics, Plesk/cPanel multi-pool support, ghost-site detection, web-shell scanner with defensive-code detection, ClickHouse renderer) remain intact and unchanged.

What is new in v0.47.11

Full ClickHouse integration plus a long list of UI/RCA correctness fixes captured during a live VM session.

ClickHouse:

New mod_clickhouse collector. Detection by PID + listening port, dedup across multiple listeners. Optional clickhouse-client subprocess query backed by /root/.xtop_secrets credentials (mode 0600).
18 deep metrics: active / total / failed / long-running queries, pending mutations, active merges, replication queue, replicas degraded, databases, active parts, total rows, top tables, top queries (1h), errors (5m), top error, mark / uncompressed cache hit %, async-insert queue, per-tick recommendations.
Purpose-built UI renderer (ui/page_apps_clickhouse.go). Replaces the alphabetical key=value dump with grouped sections: Health / Storage / Queries / Ingest / Caches / Replication / Errors / Recommendations. Humanized numbers (3.4 B rows, 267.5 K queries). Inline anomaly highlighting — red for >5% failed queries, <80% mark-cache hit, runaway merges, replicas degraded.
Error hints inline. WRONG_PASSWORD carries a hint pointing at local clients (Grafana datasource, automation scripts) and the server log path; TIMEOUT_EXCEEDED and MEMORY_LIMIT_EXCEEDED get their own one-line tuning hint.

Engine / RCA correctness:

Canonical context-switch rate from /proc/stat ctxt (was reporting 0/s on hosts where per-process collection skipped kernel threads).
cgroup tree walk now suppressed only at guard level ≥ 2, so L1 throttling still produces "top group" rows in the UI.
CPU Throttle Cascade pattern gated on actual cpu.cgroup.throttle evidence. New CPU Oversubscription pattern fires when run-queue is hot without cgroup throttle.
Run-queue impact estimate switched to Load1 for consistency with the load-average header.

UI:

Layout never breaks horizontally — boxRow truncates via ansi.Truncate so styled content stays inside the frame.
Removed duplicate | CPU | MEM | IO | Load | DISK | banner on Overview.
"Deep metrics paused" notice collapsed from an 8-line block to one dim line, with R bound to disable the resource guard and force a full tick on the current app page.
App Load Distribution falls back to global app instances when no cgroup rows exist; case-insensitive cgroup matching for IO attribution.

What is new in v0.47.2

MASTERS table cleanup + first round of web-shell false-positive fixes.

MASTERS section as a real table. Aligned columns (PHP / POOL / WORKERS / LISTEN / STATE) instead of one wide line per pool.
Semantic status states replace the misleading STATUS-FAIL:
- ok (green) - FastCGI status fetch worked
- no-status (dim) - pool healthy but pm.status_path not configured (Plesk + manual default - informational, not an error)
- no-socket (red) - listen address could not be resolved
- unreachable (red) - socket exists but connect failed
Web-shell scanner: defensive-code detection so plugins listing dangerous functions in their deny-lists do not trip:
- create_function needle now requires an open-paren - bare string mentions inside security plugins deny-list arrays no longer trip.
- Files in validator/sanitiz/security/firewall paths are skipped.
- Files containing $forbidden, $blacklist, $denylist, $banned_functions, or PHP classes named Validator/Sanitizer/Firewall/SecurityCheck are recognized as defensive code and skipped.

What's new in v0.47.1

Generic PHP-FPM diagnostic — works on Plesk, cPanel, aaPanel, vanilla Debian/Ubuntu/RHEL, and manual setups. v0.47.1 fixes the one-master-one-pool assumption in v0.47.0 that broke Plesk (which uses one master with N pools, one per site, plus relative socket paths).

Multi-pool per master. Plesk-style: one master can host 20+ pools, each with its own listen socket. Each pool surfaces as its own per-site row.
Relative listen-path resolution. When a pool's listen= is a bare name (Plesk: php-fpm.sock), resolved against /var/www/vhosts/system/<pool>/, <chdir>/, /run/, /var/run/, /tmp/ — first existing path wins.
Worker → pool attribution. Workers' cmdline (php-fpm: pool <name>) bucketed into the right pool. Worker count shown even when FastCGI status is off.
PHP version inference from FPM pool or vhost enable-php-XX include.
Panel-specific vhost paths. Plesk (/etc/nginx/plesk.conf.d/vhosts, /var/www/vhosts/system/<dom>/), cPanel (/etc/userdatadomains), aaPanel/BT, Debian/Ubuntu — all auto-discovered.
Better dedup. www.example.com + example.com server_names no longer produce two apps. Plesk's internal plesk-php8X-fpm.plesk-service.* service pools filtered out.

What's new in v0.47.0

Full PHP-FPM per-site diagnostic. New F8 TUI page + xtop phpfpm cmd. Answers "which app is each php-fpm worker actually running, who's hitting it, and what's slow?" — the gap in every existing PHP-FPM monitoring tool.

Auto-discovery. Finds every running php-fpm master via /proc (no config needed), parses each pool's pm.status_path + listen socket. Works on aaPanel/BT (/www/server/php/*/etc/), Debian/Ubuntu (/etc/php/*/fpm/), RHEL, custom installs.
Per-worker view via FastCGI status. Queries pm.status_path?full over each pool's Unix socket (~2 ms per pool). Surfaces each worker's PID, state, current script, request URI, last CPU%, last memory. Joins with /proc/<pid>/{stat,io} for live CPU%/RSS/disk I/O.
Per-site aggregation. Buckets workers by docroot — wp-cron.php on new.dula.ai vs index.php on dev.fairwayexchange.com — so the pool www ambiguity goes away.
Access-log analysis. Incremental tail of each site's access log. Top client IPs with reverse-DNS + cloud-provider tag (AWS, Hetzner, OVH, DigitalOcean, Cloudflare, Tor exit, UptimeRobot, ~20 more). Hot URIs with scanner-probe markers (/.env, /legacy.php). WHO HIT WHAT table — top IP/URL combinations for IR triage.
Slow-log analysis. Tails /var/log/php-fpm.log / /www/server/php/*/var/log/slow.log. Top slow scripts with full path. BLOCKING PHP CALLS panel — every function in the slow-log stack is classified as [critical] / [heavy] / [normal] with an explanation and optimization tip (Elementor render hooks, opcache config, hook-system overhead, DB query patterns, web-shell call signatures).
Filesystem scan. Walks the docroot looking for web-shell signatures (eval/shell_exec/system on user input, FilesMan/WSO/b374k/c99 patterns, packed base64 payloads) and unexpected ELF/shebang binaries. Cached 10 min, on-demand with D.
Ghost-site detection. When nginx still has a vhost but the docroot is deleted, surfaces it as [GHOST SITE — docroot missing] with the cleanup commands (rm vhost.conf; nginx -s reload) ready to paste. Stops the 404 noise.
8 new RCA rules. phpfpm.webshell.suspect, phpfpm.fs.webshell, phpfpm.fs.binary, phpfpm.bruteforce.detected, phpfpm.vhost.stale, phpfpm.single_ip.dominant, phpfpm.5xx.surge, phpfpm.slow.high — each with severity + specific remediation.
Cost model. Throttled to 30 s refresh. FastCGI status fetch + log tails are always-on (~10-15 ms total). Docroot scan only runs on cold first walk or user-pressed D. Resource Guardian still in effect.

Usage:

xtop phpfpm                          # per-site dashboard
xtop phpfpm --site=fairway           # one site
xtop phpfpm --deep-scan              # force fresh docroot scan
xtop phpfpm --json                   # full data
xtop                                 # then press F8 for live TUI page

The F7 per-process-connections page from v0.46.x has been removed.

What's new in v0.39.1

Multi-host fleet hub with live web dashboard at http://hub:9898/ (xtop hub)
Fleet TUI — browse every host's RCA from one terminal (xtop fleet)
Post-mortem reports per incident (xtop pm @1)
VM right-sizing report (xtop cost) + named baselines (xtop baseline)
Runbook library — drop markdown files in ~/.xtop/runbooks/ and xtop shows the matching one inline
Auto-correlation in every incident: config drift, app logs, OTel traces, past-incident diff
Kubernetes pod-aware cgroup view (auto-detects kubepods.slice)
Confidence calibration — RCA learns from incident outcomes

Full change log: docs/CHANGES-v0.39.1.md · Complete usage guide: docs/USAGE.md

The Problem — Why traditional tools fail
How xtop Solves It — What xtop answers in 5 seconds
Features
- Root-Cause Analysis Engine — 68 evidence checks, 4 bottleneck detectors
- RCA Decision Engine — Narrative, pattern detection, temporal causality, blame
- Statistical RCA Intelligence — EWMA baselines, z-score anomaly, Pearson correlation, Holt forecasting, seasonal awareness, process profiling, golden signals, causal learning
- Health Levels — OK / Inconclusive / Degraded / Critical
- 17 Interactive Pages — Overview, CPU, Memory, IO, Network, Cgroups, Timeline, Events, Probe, Thresholds, DiskGuard, Security, Logs, Services, Diagnostics, Intel, Apps, Proxmox
- 6 Overview Layouts — Two-Column, Compact, Adaptive, Grid, htop, btop
eBPF Deep Investigation — Off-CPU, IO latency, lock contention, TCP retransmits
eBPF Network Security Intelligence — SYN flood, port scan, DNS, C2 beacon, exfiltration
Predictive Exhaustion Warnings — Memory, swap, conntrack trend prediction
Anomaly Onset Tracking — Timestamped bottleneck origins
Network Intelligence — Interface classification, TCP state analysis, conntrack
SMART Disk Health — Temperature, wear level, reallocated sectors
Incident Recording & Replay — Flight recorder for postmortem
Event Detection — Automatic incident detection and logging
Doctor Mode — Comprehensive health check (CLI, JSON, Markdown, cron, alerts)
Application Diagnostics & RCA — 15 auto-detected apps with deep health analysis
- Supported Applications — MySQL, PostgreSQL, Redis, Nginx, and 11 more
- Deep Metrics Collection — Tier 1 (process) + Tier 2 (protocol/CLI) metrics
- Credential Configuration — ~/.xtop_secrets setup
Active Service Detection — Auto-detect MySQL, Redis, Docker, K8s, etc.
Shell Health Widget — Bash/Zsh prompt integration
Cron Integration — Automated health checks
9 Output Modes — TUI, watch, doctor, JSON, Markdown, daemon, record/replay, shell widget
Quick Start — One-liner install
Installation — .deb, .rpm, Arch Linux, from source, uninstall
Documentation
- CLI Reference — All flags and options
- Key Bindings — Keyboard shortcuts
- Data Sources — 20+ Linux kernel interfaces
- Examples — Usage examples
Configuration

The Problem

You're on-call. A page fires at 3 AM. Your server is slow. Now what?

You open htop...           → 200 processes. Which one? Why?
You run top...             → CPU is 80%. So what? Is it user? system? steal? throttled?
You check iostat...        → Disk util 95%. Which cgroup? Which process? Is it the root cause or a symptom?
You run vmstat...          → Pages swapping. But why? Who allocated all the memory?
You try dmesg...           → OOM killer fired. 10 minutes ago. You're reading the past.
You open 6 more terminals... → ss, nstat, sar, perf, dstat, iotop...

45 minutes later, you're still correlating data from 12 different tools, none of which talk to each other, none of which tell you the actual root cause, and none of which tell you what to do about it.

This is broken.

Traditional Linux monitoring tools were designed in the 1990s for a world where one admin watched one server. They show you raw numbers and expect you to be the correlation engine. In production, that's not monitoring — that's archaeology.

How xtop Solves It

xtop replaces the entire traditional troubleshooting workflow with a single command:

sudo xtop

Instead of 50+ fragmented tools, xtop gives you one unified console that:

Traditional Approach	xtop Approach
Raw metrics across 12+ tools	Unified dashboard — all subsystems in one view
You manually correlate signals	Automatic RCA engine — correlates CPU, memory, IO, network, cgroups
"CPU is 80%"	"CPU Contention: api-server throttled 34% in user.slice — Confidence: 72%"
Read man pages for thresholds	Evidence-based scoring — 68 evidence checks, weighted formulas
Guess who's causing it	Culprit attribution — pinpoints the cgroup and PID responsible
Google for remediation	Actionable suggestions — tells you what commands to run
Lose context between snapshots	Anomaly tracking — "Started 62s ago, triggered by IO PSI spike"
No history, no replay	Flight recorder — record incidents, replay them later for postmortem
Plaintext stdout	Incident reports — one-command Markdown export for tickets

What xtop Answers in Under 5 Seconds

✦ What is the bottleneck?     → IO Starvation (score: 78%)
✦ Who is causing it?          → mysql (PID 14923) in db.slice
✦ How confident are we?       → 85% — 4/4 evidence groups confirmed
✦ How long has this been?     → Started 2m 14s ago, triggered by IO PSI full spike
✦ What should I do?           → iotop -oP | Check cgroup io.max for db.slice
✦ What's the causal chain?    → sda await 45ms → IO PSI full 12% → 3 D-state → load 8.2 → app latency risk
✦ What's next at risk?        → Memory at 78% — exhaustion in ~22 minutes at current trend

Features

Root-Cause Analysis Engine

The heart of xtop. Four parallel bottleneck detectors continuously score system health using 68 evidence checks across 4 independent signal groups:

Bottleneck	Evidence Groups	What It Detects
IO Starvation	PSI, D-state tasks, Disk latency/util, Dirty pages	Storage-bound workloads, saturated disks, writeback storms
Memory Pressure	PSI, Available RAM, Swap activity, Direct reclaim, Major faults, OOM kills	Memory exhaustion, cache thrashing, swap storms, OOM events
CPU Contention	PSI, Run queue depth, Context switches, Cgroup throttling, CPU steal	Overcommitted CPUs, throttled containers, noisy neighbors, stolen cycles
Network Overload	Packet drops, TCP retransmits, Conntrack pressure, SoftIRQ overhead, TCP state anomalies, Errors	Saturated NICs, connection leaks, firewall table exhaustion

Trust Gating: A bottleneck is only reported when 2+ independent evidence groups confirm it. This eliminates false positives from single-metric spikes. Confidence scales from 30% (2 groups) to 98% (5+ groups).

RCA Decision Engine (v0.39.1)

Beyond raw signals, xtop's decision engine tells you EXACTLY what's wrong, why, what caused it, and what to do:

Narrative Engine — Human-readable root cause explanations replace raw metric names. Instead of "CPU Contention" you see "CPU throttle cascade — cgroup limits saturating run queue" with top evidence lines and impact summary
Pattern Detection — 32 named failure patterns (OOM Crisis, Memory-Induced IO Storm, CPU Throttle Cascade, Disk IO Saturation, VM Noisy Neighbor, Network Congestion, Socket Leak, Conntrack Exhaustion, DDoS SYN Flood, Port Scan Attack, C2 Beacon Active, Data Exfiltration, Slab Leak, IRQ Imbalance, and more) checked by priority
Temporal Causality — Tracks signal onset times to identify which signal fired first and builds chains like retransmits (T+0s) → drops (T+3s) → threads blocked (T+12s)
Blame Attribution — Top offending processes per bottleneck domain with process-specific metrics (cpu%, threads, ctxsw, mem%, RSS, IO MB/s, CLOSE_WAIT count)
Security Evidence — BPF sentinel and watchdog probes feed security-specific evidence (SYN flood, port scan, lateral movement, data exfiltration, DNS tunneling, C2 beacon) into the RCA scoring with dedicated threat score bypass
Statistical Intelligence — EWMA baselines, z-score anomaly detection, Pearson cross-metric correlation, Holt double-exponential trend forecasting, seasonal hour-of-day awareness, per-process behavior profiling, and causal strength learning — all pure math, zero external dependencies
Application-Level RCA — 15 auto-detected application modules with deep health diagnostics. Each module scores health from 100 down, applying weighted penalties for degraded metrics (e.g., MySQL buffer pool hit ratio < 95% = -15, PostgreSQL deadlocks > 0 = -10, HAProxy servers DOWN = -15 each). Total of 120+ application health rules across all modules correlating internal app state with system-level bottlenecks

Press e (Explain) to see the full ROOT CAUSE → EVIDENCE → IMPACT → TEMPORAL CAUSALITY → TOP OFFENDERS breakdown. Press Y to see per-application health diagnostics with deep metrics.

Statistical RCA Intelligence (v0.39.1)

xtop doesn't just check thresholds — it learns your system's normal behavior and detects anomalies that static rules would miss. Eight statistical modules run continuously with zero configuration:

Module	What It Does	How It Works
EWMA Baselines	Learns dynamic "normal" for every metric	Exponentially Weighted Moving Average with Welford's online variance — adapts to drift, flags deviations > 3σ
Z-Score Anomaly	Detects relative anomalies in sliding windows	60-sample sliding window z-score — catches sudden spikes even when absolute values are normal
Pearson Correlation	Discovers cause-effect relationships between metrics	Streaming Pearson R across 20 pre-defined metric pairs — surfaces correlations
Holt Forecasting	Predicts where metrics are heading	Double exponential smoothing with trend — "Memory exhaustion in ~22 minutes" with ETA-to-threshold
Seasonal Awareness	Learns recurring patterns by hour-of-day	Per-hour EWMA baselines suppress alerts for known patterns — "CPU always high at 2AM during backups"
Process Profiling	Detects when a process deviates from its own baseline	Per-Comm EWMA for CPU, memory, IO — flags when `mysql` suddenly uses 3x its normal CPU
Golden Signals	Google SRE signal approximation from /proc data	Latency (IO PSI + await), Traffic (net bytes + disk IOPS), Errors (retransmits + drops + OOM), Saturation (run queue + mem pressure + conntrack)
Causal Learning	Blends observed causality with hardcoded rules	Tracks rule prediction accuracy, blends 70% hardcoded + 30% observed weight (after 20+ observations)

Key design:

Pure math — zero external dependencies, no ML frameworks, no Python, no GPUs
Online algorithms — constant memory, O(1) per update, no batch processing
100-sample warmup — baselines are silent until statistically meaningful
Minimum stddev floor — prevents false anomalies on perfectly stable metrics
Thread-safe — all trackers use sync.RWMutex for concurrent access

Health Levels

Level	Meaning	Action
OK	All evidence groups inactive	System healthy at 95% confidence
INCONCLUSIVE	Signals present but fewer than 2 evidence groups confirm	Possible issue — run eBPF probe for deeper investigation
DEGRADED	Score 25-59%, 2+ evidence groups	Active bottleneck identified — investigate culprit
CRITICAL	Score 60%+, 2+ evidence groups	Severe bottleneck — immediate action required

17 Interactive Pages

Key	Page	What You See
`0`	Overview	Health banner, PSI pressure bars, capacity headroom, resource owners, causal chain, RCA scores, trend sparklines
`1`	CPU	Utilization breakdown (user/sys/iowait/steal/softirq), cgroup CPU rankings, throttle detection, per-process CPU table
`2`	Memory	Full 13-category memory breakdown, active/inactive pages, swap status, vmstat counters, hugepages, cgroup + process memory rankings
`3`	IO	Per-device performance table (MB/s, IOPS, await, util%, queue depth), IO type analysis (sequential/random), raw counters, SMART disk health, D-state tracking
`4`	Network	Health verdict, aggregate throughput, TCP connection state distribution with visual bars, per-interface table with link state/speed/type/master detection, protocol health (TCP/UDP), conntrack usage, top consumers, kernel SoftIRQ overhead
`5`	Cgroups	Full sortable table of all cgroups — sort by CPU%, throttle%, memory, OOM kills, IO rate. Auto-detects cgroup v1/v2/hybrid
`6`	Timeline	5-minute rolling ASCII sparkline charts — 16 time series across CPU, memory, IO, network
`7`	Events	Automatically detected incidents with timestamps, duration, peak scores, bottleneck type, culprit attribution
`8`	Probe	Real-time eBPF investigation results — off-CPU analysis, IO latency histograms, lock contention, TCP retransmit tracking
`9`	Thresholds	Live view of all RCA threshold values vs current readings — see exactly which checks are passing/failing
`D`	DiskGuard	Filesystem space monitor with auto-contain — SIGSTOP/SIGCONT top disk writers when mounts cross critical thresholds
`L`	Security	eBPF network security intelligence — 14 collapsible sections with threat detection, attack analysis, flow intelligence
`O`	Logs	Live system log viewer with filtering
`H`	Services	Active service health monitoring
`W`	Diagnostics	System diagnostics and troubleshooting
`X`	Intel	Impact scores, cross-signal correlation, runtime detection, SLO status, autopilot actions, incident history
`Y`	Apps	Application diagnostics — auto-detected MySQL, PostgreSQL, Redis, Nginx, Apache, HAProxy, PHP-FPM, MongoDB, Memcached, RabbitMQ, Kafka, Elasticsearch, Docker, Caddy, Traefik with deep health RCA
`Z`	Proxmox	Proxmox VE host dashboard — host CPU/RAM/load/PSI, network interfaces, disk IO/SMART health, VM status table, per-VM details, storage pools (auto-detected, hidden on non-PVE hosts)

6 Overview Layouts

Switch instantly with v / V / F1-F6:

Layout	Style	Best For
Two-Column (F1)	Subsystems left, owners + chain right	Daily monitoring, wide terminals
Compact (F2)	Dense single-column summary	Narrow terminals, quick glance
Adaptive (F3)	Healthy=1 line, unhealthy=expanded	Busy systems with mixed health
Grid (F4)	2x2 subsystem dashboard	Executive overview, presentations
htop (F5)	htop-style process list	Familiar process monitoring
btop (F6)	btop-style resource dashboard	System resource overview

Press Ctrl+D to save your preferred layout as the default.

eBPF Deep Investigation

When the RCA engine identifies a bottleneck but you need process-level proof, press I to launch a 10-second eBPF probe. Four kernel-level packs run simultaneously:

Probe Pack	Kernel Tracepoint	What It Reveals
Off-CPU Analysis	`sched_switch`	Which processes are being forced off-CPU, for how long, and why (futex lock, disk IO, network wait, epoll)
IO Latency	`block_rq_issue` + `block_rq_complete`	Per-device I/O latency distribution with P50/P95/P99 percentiles — identifies slow disks and heavy writers
Lock Contention	`sys_enter_futex` + `sys_exit_futex`	Which processes are blocked on mutex/futex locks, total wait time, contention hotspots
TCP Retransmits	`tcp_retransmit_skb`	Per-process retransmit counts with destination IP:port — identifies flaky network paths

Key design decisions:

Pure Go implementation via cilium/ebpf — no CGo, no clang at runtime
Filters out kernel threads, idle daemons, and the monitoring process itself
Off-CPU probe tracks only involuntary off-CPU time (preemption + D-state), not voluntary sleep
Graceful degradation: if one probe fails to attach, the others continue
Results boost RCA confidence when they corroborate the detected bottleneck

eBPF Network Security Intelligence (v0.21.0+)

Press L to open the Security Monitor — a dedicated page with 14 collapsible sections powered by always-on BPF sentinel probes and auto-triggered watchdog probes.

Sentinel Probes (always-on, zero-config):

Probe	Kernel Hook	What It Detects
SYN Flood	`tcp_conn_request`	Distributed SYN floods — per-source-IP SYN rate and half-open ratio
Port Scan	`tcp_v4_send_reset`	Sequential/randomized port scans — unique port diversity from single source
DNS Monitor	`udp_sendmsg` + `udp_recvmsg`	DNS query anomalies — high query rates, unusual query lengths
Connection Rate	`inet_sock_set_state`	Lateral movement — processes connecting to unusually many unique destinations
Outbound Volume	`tcp_sendmsg`	Data exfiltration — large sustained outbound transfers to external IPs

Watchdog Probes (auto-triggered when sentinels detect anomalies):

Probe	Attachment	What It Inspects
TCP Flags	TC ingress classifier	XMAS, NULL, SYN+FIN — crafted packet detection
DNS Deep	TC ingress classifier	DNS tunneling indicators — TXT ratio, query length entropy
TLS Fingerprint	TC ingress classifier	JA3 fingerprinting — detect known C2 framework TLS signatures
Beacon Detect	`tcp_sendmsg`	C2 beacon detection — periodic low-jitter connection patterns

Security Page sections: SSH/Auth, Listening Ports, SUID Anomalies, Process Executions, Ptrace Detection, Reverse Shells, Fileless Processes, Kernel Module Loads, Network Threat Overview, Attack Detection, DNS Intelligence, Flow Intelligence, TLS/Beacon Analysis.

Smart filtering: Loopback (127.x), private IPs (10.x, 172.16-31.x, 192.168.x), and xtop's own PID are automatically excluded. Proxy-aware thresholds prevent false positives on forward proxy servers.

Predictive Exhaustion Warnings

xtop doesn't just tell you what's wrong now — it predicts what will go wrong next:

Resource	What It Tracks	Alert
Memory	MemAvailable trend over 60 samples	"Memory exhaustion in ~22 minutes at current trend"
Swap	Swap usage growth rate	"Swap full in ~8 minutes"
Conntrack	Connection table growth	"Conntrack table full in ~15 minutes"

Requires 30+ seconds of history. Fires when predicted exhaustion is under 60 minutes.

Anomaly Onset Tracking

Every bottleneck is timestamped to its origin:

IO Starvation — Score: 72% — Started 2m 14s ago
  Triggered by: IO PSI full avg10 crossed 5% → currently 12.3%
  Culprit: mysql (PID 14923) — top consumer since 1m 48s
  Biggest change (30s): IO PSI full +8.2 percentage points

This eliminates the "how long has this been happening?" question that plagues traditional monitoring.

Network Intelligence

xtop's network page goes far beyond ifconfig or ip -s:

Interface classification: Automatically identifies physical, virtual, bridge, bond, veth, VLAN, tunnel, and WiFi interfaces
Bridge/bond slave detection: If enX1 shows 0 traffic because it's a bridge slave, xtop shows: └─ slave of br0 — traffic counters may be on master
Link state & speed: Shows UP/DOWN status and negotiated speed for every interface
TCP state analysis: Visual bars for all 9 connection states with anomaly thresholds (TIME_WAIT>5K, CLOSE_WAIT>100)
Conntrack monitoring: Table usage percentage with exhaustion prediction
Protocol health: TCP retransmit rate, UDP buffer errors, segment rates, SoftIRQ overhead

SMART Disk Health

Page 3 (IO) includes physical disk health via smartctl:

Health status (PASSED/FAILED)
Temperature with thermal thresholds
Wear level percentage (SSD/NVMe)
Reallocated sector count (early failure indicator)
Pending sector count (active failure indicator)
Power-on hours
Model identification

Incident Recording & Replay

Record a live session as a flight recorder:

sudo xtop -record /var/log/xtop-incident.wlog

Replay it later for postmortem analysis — no root required:

xtop -replay /var/log/xtop-incident.wlog

Every snapshot is preserved with full fidelity: metrics, rates, RCA results, evidence checks, causal chains. Review exactly what the system looked like during the incident.

Event Detection

xtop automatically detects and logs incidents:

Debounced transitions: 3 consecutive non-OK ticks required to open an event (prevents flapping)
Per-event tracking: Peak health level, peak RCA score, bottleneck type, culprit process/cgroup
Metrics captured: Peak CPU%, peak memory%, peak IO PSI for each incident
Persistent logging: Events written to ~/.xtop/events.jsonl in daemon mode
Full audit trail: Start time, end time, duration, evidence, causal chain

Doctor Mode

Comprehensive health check that scans every subsystem in one shot:

sudo xtop -doctor                    # Beautiful aligned CLI report
sudo xtop -doctor -watch             # Auto-refreshing (like top)
sudo xtop -doctor -watch -interval 5 -count 3  # 3 iterations at 5s
sudo xtop -doctor -json              # Machine-readable JSON
sudo xtop -doctor -md                # Markdown for tickets
sudo xtop -doctor -cron              # Cron-friendly (silent if OK, exit codes)
sudo xtop -doctor -alert             # Send alerts on state changes

What it checks:

Category	Checks
CPU	Utilization, load average (with IO-blocked decomposition), PSI pressure
Memory	Usage %, swap %, PSI, absolute available threshold
Disk	Per-mount usage, DiskGuard state, per-device latency/util, inode usage, PSI IO
Network	Overall health, TCP retransmits, drops, conntrack, CLOSE_WAIT leaks
System	File descriptors, systemd failed units, NTP sync, security updates pending
Security	Fileless process detection with forensic detail (exe, cmd, cwd, RSS, FDs, network connections)
Docker	Disk usage, container health
SSL	Let's Encrypt certificate expiration
Services	Auto-detected active services with deep health checks (MySQL, PostgreSQL, Redis, Docker, K8s, WireGuard)

Alert dispatch: Supports webhooks, Slack, Telegram, email, and custom commands. Only fires on state changes (OK→WARN, WARN→CRIT, etc.) to prevent alert fatigue.

Exit codes: 0 = OK, 1 = warnings, 2 = critical — integrate directly into monitoring pipelines.

Application Diagnostics & RCA

Press Y to open the Apps page — xtop auto-detects 15 applications and runs deep root-cause analysis on each. No agents, no plugins, no configuration required. If the process is running, xtop finds it and starts collecting.

How It Works

The RCA engine uses a two-tier collection strategy:

Tier 1 (always available) — Process-level metrics from /proc: RSS memory, thread count, file descriptors, TCP connections, uptime. Works for every detected app with zero configuration.
Tier 2 (deep metrics) — Protocol-level collection via native APIs, CLI tools, or raw protocol:
- MySQL: SHOW GLOBAL STATUS (27 key variables), SHOW PROCESSLIST, SHOW ENGINE INNODB STATUS, SHOW REPLICA STATUS
- PostgreSQL: pg_stat_activity, pg_stat_database, pg_stat_bgwriter, dead tuples, lock contention, replication lag
- Redis: Raw RESP protocol INFO ALL + SLOWLOG GET — 80+ metrics including memory, latency percentiles (p50/p99/p99.9), replication lag, keyspace, persistence, command stats, fork timing, client capacity, plus workload-driven recommendations engine
- Elasticsearch: REST API — cluster health, shard status, JVM heap, index counts, node stats
- MongoDB: mongosh — serverStatus, WiredTiger cache, opCounters, lock queues, replication lag
- Nginx: stub_status HTTP endpoint — active connections, accepts/handled/requests, reading/writing/waiting
- Apache: mod_status endpoint — scoreboard analysis, request rates, worker utilization, MPM detection
- HAProxy: Unix socket show stat/show info/show sess — CSV stats parsing, backend health, session rates, queue depth, latency breakdown (queue/connect/response/total), slow backend spotlight, retry & redispatch analysis, denied request tracking, peak vs current sessions, config warnings, server state change tracking, per-direction TCP state analysis, top IP breakdown, connection error RCA with per-backend blame
- PHP-FPM: Pool config parsing + per-worker state analysis via /proc/PID/stat — active/idle workers, utilization, memory tracking
- RabbitMQ: Management API — messages, queues, node resources, memory/disk alarms, consumer health
- Memcached: Raw TCP stats/stats slabs — hit ratio, evictions, slab analysis, memory pressure
- Kafka: Config parsing + CLI tools — broker ID, topic/group counts, log dir usage, JVM hsperfdata
- Caddy: Admin API (:2019) — live config, upstream health, metrics
- Traefik: API (:8080) — routers/services/middlewares, entrypoints, health check
- Docker: Unix socket — containers, images, networks, disk usage, health checks

Supported Applications

Application	Detection	Deep Metrics	Health Checks	Credentials
MySQL / MariaDB	`mysqld`, `mariadbd`	InnoDB buffer pool, slow queries, replication, deadlocks, lock waits, temp tables	17 rules	Required
PostgreSQL	`postgres` (postmaster)	Cache hit ratio, dead tuples, vacuum lag, blocked queries, bgwriter, replication	12 rules	Required
Redis	`redis-server`	Memory, latency p50/p99/p99.9, hit ratio, evictions, fragmentation, RSS ratio, persistence, fork timing, replication lag, keyspace, command stats, slow log, client capacity, pub/sub, workload recommendations	16 rules	If AUTH enabled
Elasticsearch	Java + ES cmdline	Cluster health, shards, JVM heap, indices, node stats	8 rules	If X-Pack enabled
MongoDB	`mongod`, `mongos`	WiredTiger cache, opCounters, lock queues, replication lag, slow ops	10 rules	If auth enabled
Nginx	`nginx` (master)	Active connections, request rate, dropped connections, worker state, scoreboard	7 rules	No
Apache	`httpd`, `apache2`	Scoreboard, worker utilization, request rate, MPM config, CPU load	8 rules	No
HAProxy	`haproxy` (master)	Backend server health, session rates, queue depth, 5xx rate, CPU idle, latency breakdown, slow backends, retry/redispatch, denied requests, peak vs current, config warnings, state changes, TCP direction analysis	11 rules	No
PHP-FPM	`php-fpm*` (master)	Worker utilization, max_children saturation, per-worker memory, pool config	7 rules	No
RabbitMQ	`beam.smp` + rabbit	Message backlog, unacked, memory/disk alarms, queue health, node resources	11 rules	Default guest/guest
Memcached	`memcached`	Hit ratio, evictions, slab waste, memory pressure, connection rejection	9 rules	No
Kafka	Java + kafka cmdline	Broker config, topic count, consumer groups, log dir size, JVM hsperfdata	4 rules	No
Caddy	`caddy`	Live config, upstream health, TLS, Caddyfile analysis	Basic	No
Traefik	`traefik`	Routers/services, health check, entrypoints, error rates	3 rules	No
Docker	`dockerd`	Containers, images, networks, disk usage, health diagnostics	5 rules	No

Deep Metrics Collection

Each application module follows the same RCA pattern:

Detect — Scan process list every tick, match by comm name and cmdline
Collect Tier 1 — Read /proc/PID/{status,stat,fd} + /proc/net/tcp for connections
Collect Tier 2 — Query the app via its native protocol/CLI/API (with timeout, graceful fallback)
Compute — Derive ratios (cache hit%, connection usage%, lock contention%, etc.)
Score Health — Start at 100, apply weighted penalties per issue, clamp to [0, 100]
Report Issues — Each penalty generates a human-readable health issue with context

The health score drives a diagnostic badge: OK (80-100), WARN (50-79), CRIT (0-49).

Credential Configuration

Apps that need authentication show a CREDENTIALS REQUIRED notice at the top of their detail page with the exact JSON template to copy.

Create ~/.xtop_secrets (JSON format, chmod 600):

{
  "mysql": {
    "host": "127.0.0.1",
    "port": 3306,
    "user": "root",
    "password": "YOUR_MYSQL_PASSWORD"
  },
  "postgresql": {
    "host": "127.0.0.1",
    "port": 5432,
    "user": "postgres",
    "password": "YOUR_PG_PASSWORD",
    "dbname": "postgres"
  },
  "mongodb": {
    "uri": "mongodb://user:password@127.0.0.1:27017/admin"
  },
  "redis": {
    "host": "127.0.0.1",
    "port": 6379,
    "password": "YOUR_REDIS_PASSWORD"
  },
  "rabbitmq": {
    "host": "127.0.0.1",
    "port": 15672,
    "user": "guest",
    "password": "guest"
  },
  "elasticsearch": {
    "url": "http://127.0.0.1:9200",
    "user": "elastic",
    "password": "YOUR_ES_PASSWORD"
  }
}

Only include the apps you use. Apps without authentication (Nginx, Apache, HAProxy, PHP-FPM, Memcached, Kafka, Caddy, Traefik, Docker) work automatically with zero configuration.

Active Service Detection

Doctor mode auto-detects running services and runs health checks — no pre-configuration needed:

Service	Health Check
sshd	Process running
nginx / apache / caddy	Process running, stub_status/server-status check
MySQL / MariaDB	Process running, SHOW GLOBAL STATUS analysis
PostgreSQL	Process running, pg_stat_database analysis
Redis	`INFO` command via RESP protocol
Docker	Container count, unhealthy/restarting detection
Kubernetes	kubelet running, `kubectl get nodes` status
WireGuard	Interface active (`wg0`, etc.)
HAProxy	Stats socket analysis, backend health, latency breakdown, config warnings
MongoDB	serverStatus via mongosh
RabbitMQ	Management API health
Elasticsearch	Cluster health API
Memcached	`stats` command via TCP
Kafka	Process running, broker config
PHP-FPM	Pool utilization, worker state
DNS (named/dnsmasq/unbound)	Process running

Services that aren't installed are silently skipped — only active services appear in the report.

Add system health to your shell prompt:

# Bash
eval "$(xtop -shell-init bash)"

# Zsh
eval "$(xtop -shell-init zsh)"

# Tmux status bar
xtop -tmux-status

Shows a colored health indicator (OK/WARN/CRIT) that updates from the daemon's last health check.

Cron Integration

# Print crontab line for automated health checks
xtop -cron-install

# Example cron entry (runs every 5 minutes, alerts on state change)
*/5 * * * * /usr/local/bin/xtop -doctor -cron -alert 2>/dev/null

9 Output Modes

Mode	Command	Use Case
Interactive TUI	`sudo xtop`	Live monitoring and investigation
Watch Mode	`sudo xtop -watch -section cpu`	Headless CLI output, SSH-friendly
Doctor	`sudo xtop -doctor`	Comprehensive health check report
Doctor Watch	`sudo xtop -doctor -watch`	Auto-refreshing health checks (like `top`)
JSON Export	`sudo xtop -json \| jq`	Scripting, alerting, integrations
Markdown Report	`sudo xtop -md > incident.md`	Jira/Slack/GitHub ticket attachment
Daemon Mode	`sudo xtop -daemon &`	Background collection + event logging
Record/Replay	`sudo xtop -record file`	Flight recorder for postmortem
Shell Widget	`eval "$(xtop -shell-init bash)"`	System health in your bash/zsh prompt

Quick Start

One-liner Install

# Ubuntu/Debian (amd64)
wget https://github.com/ftahirops/xtop/releases/download/v0.49.0/xtop_0.49.0-1_amd64.deb
sudo dpkg -i xtop_0.49.0-1_amd64.deb

# RHEL/Rocky/Fedora (x86_64)
wget https://github.com/ftahirops/xtop/releases/download/v0.49.0/xtop-0.49.0-1.x86_64.rpm
sudo rpm -i xtop-0.49.0-1.x86_64.rpm

# Arch Linux
git clone https://github.com/ftahirops/xtop.git
cd xtop/packaging/archlinux && makepkg -si

Build from Source

git clone https://github.com/ftahirops/xtop.git
cd xtop
CGO_ENABLED=0 go build -ldflags="-s -w -X github.com/ftahirops/xtop/cmd.Version=0.49.0" -o xtop .
sudo install -m 755 xtop /usr/local/bin/xtop

Run

sudo xtop              # Full TUI, 3s refresh
sudo xtop 5            # 5-second intervals
sudo xtop -watch       # CLI mode, no TUI
sudo xtop -doctor      # Health check report
sudo xtop -json | jq   # JSON for scripting

Installation

Requirements

Requirement	Details
OS	Linux (kernel 4.20+ recommended for PSI)
Architecture	amd64 (x86_64)
Cgroups	v1, v2, or hybrid (auto-detected)
Terminal	Unicode support (for sparklines and box drawing)
Permissions	Root recommended for full `/proc/*/io` access and eBPF probes
eBPF (optional)	Kernel with BTF (`/sys/kernel/btf/vmlinux`) + root for probe packs

From .deb Package (Ubuntu 22.04/24.04, Debian)

wget https://github.com/ftahirops/xtop/releases/download/v0.49.0/xtop_0.49.0-1_amd64.deb
sudo dpkg -i xtop_0.49.0-1_amd64.deb

From .rpm Package (Rocky Linux, RHEL, AlmaLinux, Fedora)

wget https://github.com/ftahirops/xtop/releases/download/v0.49.0/xtop-0.49.0-1.x86_64.rpm
sudo rpm -i xtop-0.49.0-1.x86_64.rpm

Arch Linux (PKGBUILD)

# Build and install from PKGBUILD
git clone https://github.com/ftahirops/xtop.git
cd xtop/packaging/archlinux
makepkg -si

# Or with an AUR helper (once published to AUR)
# yay -S xtop

Builds from source automatically. Optional dependencies: nvidia-utils (GPU monitoring), docker (container name resolution).

From Source

git clone https://github.com/ftahirops/xtop.git
cd xtop
CGO_ENABLED=0 go build -ldflags="-s -w -X github.com/ftahirops/xtop/cmd.Version=0.49.0" -o xtop .
sudo install -m 755 xtop /usr/local/bin/xtop

Uninstall

sudo dpkg -r xtop        # Debian/Ubuntu
sudo rpm -e xtop          # RHEL/Rocky
sudo pacman -R xtop       # Arch Linux
# or
sudo rm /usr/local/bin/xtop

Documentation

CLI Reference

xtop [OPTIONS] [INTERVAL]

Modes:
  (default)         Interactive TUI (fullscreen)
  -watch            CLI output mode — prints to terminal with auto-refresh
  -json             Single JSON snapshot to stdout, then exit
  -md               Single Markdown incident report to stdout, then exit
  -daemon           Background collector (writes events to datadir)
  -doctor           Comprehensive health check report
  -version          Print version and exit
  -update           Check GitHub for latest release and install it

Doctor Options:
  -doctor -watch    Auto-refreshing doctor (like top/watch)
  -doctor -json     Health check as JSON
  -doctor -md       Health check as Markdown table
  -cron             Cron-friendly output (silent if OK, exit codes 0/1/2)
  -alert            Send alert on health state change
  -cron-install     Print crontab line for automated health checks

Shell Widget:
  -shell-init SHELL Output shell init script (bash or zsh)
  -tmux-status      Output tmux-formatted status segment

Options:
  -interval N       Collection interval in seconds (default: 3)
  -history N        Snapshots to keep in ring buffer (default: 600)
  -section NAME     Section for -watch mode (overview,cpu,mem,io,net,cgroup,rca)
  -count N          Iterations for -watch and -doctor -watch (0 = infinite)
  -datadir PATH     Data directory for daemon mode (default: ~/.xtop/)
  -record FILE      Record snapshots to file during TUI session
  -replay FILE      Replay recorded file through TUI (no root needed)
  -prom             Enable Prometheus metrics endpoint
  -prom-addr ADDR   Prometheus listen address (default: 127.0.0.1:9100)
  -alert-webhook URL  Webhook URL for alert notifications
  -alert-command CMD  Command to execute on alert notifications

Key Bindings

Key	Action
`0` - `9`	Switch to page (Overview, CPU, Memory, IO, Network, Cgroups, Timeline, Events, Probe, Thresholds)
`L`	Security Monitor — eBPF network threat detection
`O`	System Logs viewer
`H`	Active Services health
`W`	Diagnostics page
`X`	Intel page
`Z`	Proxmox dashboard (auto-detected)
`Y`	Application diagnostics — 15 auto-detected apps with deep RCA
`D`	Open DiskGuard page
`b` / `Esc`	Back to Overview
`j` / `k`	Scroll down / up
`g` / `G`	Jump to top / Jump down
`v` / `V`	Cycle overview layout forward / backward
`F1` - `F4`	Direct layout selection
`Ctrl+D`	Save current layout as default
`I`	Start 10-second eBPF probe investigation
`Tab`	Navigate collapsible sections (Security, Network, Probe pages)
`Enter`	Expand/collapse selected section
`A` / `C`	Expand all / Collapse all sections
`E`	Toggle Explain side panel — metric glossary for current page
`a`	Toggle auto-refresh (pause/resume)
`n`	Step one frame (replay mode while paused)
`P`	Export page to Markdown file
`S`	Save RCA snapshot to JSON file
`s`	Cycle sort column (Cgroups page)
`?`	Toggle help overlay
`q` / `Ctrl+C`	Quit

Data Sources

xtop reads from 20+ Linux kernel interfaces — no agents, no daemons, no external dependencies:

Source	Metrics
`/proc/pressure/*`	PSI (Pressure Stall Information) — the most important signal modern Linux provides
`/proc/stat`	CPU time breakdown across all states
`/proc/loadavg`	Load averages and runnable task count
`/proc/meminfo`	30+ memory metrics (anon, cache, slab, shmem, mapped, hugepages, etc.)
`/proc/vmstat`	Page faults, reclaim, swap, OOM, THP counters
`/proc/diskstats`	Per-device IO counters (reads, writes, sectors, time, queue)
`/proc/net/dev`	Per-interface packet and byte counters
`/proc/net/snmp`	TCP/UDP protocol-level counters
`/proc/net/tcp{,6}`	Per-connection TCP state tracking
`/proc/net/sockstat`	Socket allocation summary
`/proc/softirqs`	Per-CPU softirq counters
`/proc/sys/net/netfilter/*`	Conntrack table usage and limits
`/proc/sys/fs/file-nr`	File descriptor allocation
`/proc/[pid]/stat,status,io,cgroup`	Per-process CPU, memory, IO, scheduling
`/sys/fs/cgroup/`	Cgroup v1/v2 metrics (CPU, memory, IO, throttling, OOM)
`/sys/class/net/`	Interface metadata (operstate, speed, master, type)
`smartctl`	SMART disk health (temperature, wear, reallocated sectors)
eBPF tracepoints	`sched_switch`, `block_rq_*`, `futex`, `tcp_retransmit_skb`
eBPF security sentinels	`tcp_conn_request`, `tcp_v4_send_reset`, `tcp_sendmsg`, `udp_sendmsg`, `inet_sock_set_state`
eBPF security watchdogs	TC ingress classifiers (TCP flags, DNS deep, TLS fingerprint), beacon detection

Examples

# === Interactive TUI ===
sudo xtop                              # Default 3s refresh
sudo xtop 5                            # 5-second refresh interval

# === CLI Watch Mode (no TUI, SSH-friendly) ===
sudo xtop -watch                       # Overview section, 3s refresh
sudo xtop -watch -section cpu          # CPU details only
sudo xtop -watch -section io 3         # IO section, 3s interval
sudo xtop -watch -section rca          # RCA analysis only
sudo xtop -watch -section mem -count 5 # Memory, 5 iterations then exit
sudo xtop -watch -section net -interval 2

# === Doctor Health Checks ===
sudo xtop -doctor                      # One-shot health report
sudo xtop -doctor -watch               # Auto-refreshing (like top)
sudo xtop -doctor -watch -interval 5 -count 3  # 3 iterations at 5s
sudo xtop -doctor -json                # JSON output for scripting
sudo xtop -doctor -md                  # Markdown for tickets
sudo xtop -doctor -cron                # Cron-friendly (silent if OK)
sudo xtop -doctor -alert               # Alert on state change

# === Shell Health Widget ===
eval "$(xtop -shell-init bash)"        # Add to ~/.bashrc
eval "$(xtop -shell-init zsh)"         # Add to ~/.zshrc
xtop -tmux-status                      # Tmux status bar segment
xtop -cron-install                     # Print crontab line

# === Machine-Readable Output ===
sudo xtop -json | jq '.analysis.Health'
sudo xtop -json | jq '.analysis.RCA[] | select(.Score > 0)'
sudo xtop -json | jq '.analysis.PrimaryBottleneck'

# === Incident Reports ===
sudo xtop -md > /tmp/incident-$(date +%Y%m%d).md

# === Flight Recorder ===
sudo xtop -record /var/log/xtop-$(date +%Y%m%d-%H%M).wlog
xtop -replay /var/log/xtop-20260218-0300.wlog    # No root needed

# === Background Daemon ===
sudo xtop -daemon &
sudo xtop -daemon -datadir /var/lib/xtop -interval 2

# === Prometheus Exporter ===
sudo xtop -prom -prom-addr :9100
curl -s http://localhost:9100 | head

# === Alert Hooks ===
sudo xtop -daemon -alert-webhook https://example.com/xtop
sudo xtop -daemon -alert-command 'logger -t xtop \"$XTOP_EVENT\"'

Configuration

xtop loads defaults from ~/.config/xtop/config.json (or XDG_CONFIG_HOME). Use config.example.json as a starting point.

{
  "default_layout": 0,
  "interval_sec": 1,
  "history_size": 300,
  "default_section": "overview",
  "prometheus": { "enabled": false, "addr": "127.0.0.1:9100" },
  "alerts": {
    "webhook": "",
    "command": "",
    "email": "",
    "slack_webhook": "",
    "telegram_bot_token": "",
    "telegram_chat_id": ""
  }
}

Prometheus Metrics

When -prom is enabled, xtop exposes a minimal metrics set including:

Health and primary RCA score
PSI (CPU/MEM/IO)
CPU busy/user/system/iowait/steal
Memory used %, total, available
Disk max util, per-device read/write/util/await, and await histogram
Network retrans, drops/errors (global + per-interface rx/tx/util)
Top 50 cgroups by CPU (cpu/mem/io/throttle)

Alert Payloads

Alerts are emitted by both daemon mode and doctor mode (-alert) when health state changes. Supported channels: webhook, Slack, Telegram, email, and custom command. Events include: health_critical, health_ok, event_closed, and doctor_alert.

Webhook payload example:

{
  "event": "health_critical",
  "payload": {
    "bottleneck": "IO Starvation",
    "score": 78,
    "culprit": "/system.slice/docker-abc.scope",
    "process": "postgres",
    "pid": 1234
  },
  "ts": "2026-02-19T12:00:00Z"
}

Alert command environment variables:

XTOP_EVENT — event name
XTOP_PAYLOAD — JSON payload string

Demo Scenarios

Seven ready-to-run scripts simulate real-world incidents for testing and demonstration:

Script	Scenario	What xtop Detects
`demos/01-io-stall.sh`	Sync write storm with fsync	IO PSI spike, D-state tasks, disk latency, writeback pressure
`demos/02-memory-pressure.sh`	Allocate 70% of RAM	Memory PSI, low available, swap activity, direct reclaim
`demos/03-cpu-throttle.sh`	Cgroup-limited CPU burn	CPU PSI, run queue saturation, cgroup throttle %, stolen time
`demos/04-network-drops.sh`	tc netem packet loss	Packet drops, TCP retransmits, network error rate
`demos/05-port-exhaustion.sh`	TIME_WAIT connection storm	TCP state anomaly, port exhaustion prediction
`demos/06-conntrack-flood.sh`	Conntrack table exhaustion	Conntrack >80%, connection drop risk, exhaustion prediction
`demos/all-stress.sh`	All subsystems simultaneously	Multi-bottleneck detection, priority ranking, causal chains

# Terminal 1: Start the demo
sudo bash demos/01-io-stall.sh

# Terminal 2: Watch xtop detect it in real-time
sudo xtop

Why Not Just Use...

Tool	What It Does	What It Doesn't Do
`htop` / `top`	Shows process list sorted by CPU/MEM	No RCA, no correlation, no cgroup awareness, no evidence scoring, no network analysis
`iostat`	Shows disk throughput and latency	No process attribution, no PSI, no cgroup mapping, no health verdict
`vmstat`	Shows memory and swap activity	No root cause, no trend, no prediction, no actionable output
`sar`	Historical metric collection	No real-time RCA, no correlation, no TUI, requires post-processing
`dstat`	Multi-metric live output	No analysis, no evidence scoring, no bottleneck detection
`nstat` / `ss`	Network counters and sockets	No health verdict, no interface classification, no correlation with other subsystems
`perf`	Deep CPU profiling	Steep learning curve, single-subsystem, no live dashboard
`bpftrace`	Custom eBPF scripts	Requires writing programs, no built-in RCA, expert-only
xtop	All of the above, unified, with automatic RCA	One tool. One command. Full diagnosis.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        xtop TUI / CLI                           │
│  17 pages • 6 layouts • watch mode • doctor • shell widget      │
├─────────────────────────────────────────────────────────────────┤
│                    Statistical Intelligence                      │
│   EWMA Baselines • Z-Score Anomaly • Pearson Correlation         │
│   Holt Forecasting • Seasonal Awareness • Process Profiling      │
│   Golden Signals • Causal Strength Learning                      │
├─────────────────────────────────────────────────────────────────┤
│                       Analysis Engine                            │
│   RCA Scoring • Evidence Gating • Anomaly Tracking               │
│   Causal Chains • Capacity Prediction • Owner Attribution        │
│   Security Evidence • Threat Scoring • 32 Pattern Library        │
├─────────────────────────────────────────────────────────────────┤
│                       Doctor Engine                              │
│   Health checks • Service detection • SSL • Alerts • Cron        │
│   MySQL/Postgres/Redis probes • Docker • K8s • WireGuard         │
├───────────────────────┬─────────────────────────────────────────┤
│    Collector Layer     │         eBPF Probe Layer                │
│   /proc • /sys • cgroup│   Deep Dive: sched_switch • block_rq    │
│   smartctl • netfilter │     • futex • tcp_retransmit_skb        │
│   security • auth logs │   Sentinel: synflood • portscan • dns   │
│                        │     • connrate • outbound • exec • oom  │
│                        │   Watchdog: tcpflags • dnsdeep           │
│                        │     • tlsfinger • beacondetect           │
└───────────────────────┴─────────────────────────────────────────┘
              │                          │
              ▼                          ▼
     Linux Kernel (/proc, /sys)    eBPF Tracepoints + TC (BTF)

Built with:

Go — Fast, single-binary, zero runtime dependencies
Bubbletea — Terminal UI framework
Lipgloss — Styled terminal rendering (Dracula palette)
cilium/ebpf — Pure Go eBPF (no CGo, no clang at runtime)

Installed Files

/usr/local/bin/xtop                  — Binary (~17 MB, statically linked)
/usr/share/man/man1/xtop.1.gz        — Man page
/usr/share/doc/xtop/copyright         — MIT license
~/.xtop/                              — Runtime data (config, event logs, daemon state)

Contributing

Contributions are welcome. Please open an issue to discuss significant changes before submitting a PR.

git clone https://github.com/ftahirops/xtop.git
cd xtop
go build ./...
go vet ./...
sudo ./xtop

Roadmap

What's coming next for xtop:

Feature	Status	Description
Language Runtime Detection	In Progress	Auto-discover JVM, .NET, Python, Node.js, Go runtimes — zero-config JVM hsperfdata parsing, GC/heap/thread metrics, GIL-bound detection
JVM Deep RCA	In Progress	GC pause evidence, heap pressure alerts, hsperfdata binary parser — correlates JVM GC storms with CPU run queue spikes
Distributed Tracing Correlation	Planned	Correlate xtop RCA findings with OpenTelemetry spans — "this IO spike caused 200ms P99 on /api/checkout"
Kubernetes Pod RCA	Planned	Per-pod bottleneck detection with cgroup v2 mapping — "Pod X OOMKilled because Node Y memory pressure hit 85%"
GPU Monitoring	Planned	NVIDIA GPU utilization, memory, temperature via nvml — detect GPU contention for ML workloads
Anomaly Clustering	Planned	Group related anomalies into incident timelines — "these 5 metrics deviated together at T+0"
Remote Agent Mode	Planned	Lightweight agent reports to central xtop instance — fleet-wide bottleneck detection
Custom Evidence Plugins	Planned	User-defined evidence checks via YAML — "if redis.connected_clients > 10000, fire redis.client.flood"

Want to influence the roadmap? Open an issue.

License

xtop — Because the answer to "what's wrong with my server?" shouldn't take 12 tools and 45 minutes.

Documentation ¶

There is no documentation for this package.

Source Files ¶

View all Source files

main.go

Directories ¶

Path	Synopsis
api
cmd
monitor command monitor is a headless version of xtop that prints RCA results to stdout.	monitor is a headless version of xtop that prints RCA results to stdout.
xtop-agent command xtop-agent is the headless fleet-agent binary.	xtop-agent is the headless fleet-agent binary.
collector
apps
cgroup
configdrift Package configdrift provides a curated snapshot of RCA-relevant OS/kernel configuration keys for config-drift detection (Phase 4).	Package configdrift provides a curated snapshot of RCA-relevant OS/kernel configuration keys for config-drift detection (Phase 4).
dotnet
ebpf
journal
phpfpm
profiler
runtime
config
engine
replay
verifier
fleet Package fleet implements the xtop fleet hub — the central server that agents report to.	Package fleet implements the xtop fleet hub — the central server that agents report to.
identity
model
store
tests
stressors command Standalone stressor that appears with its own binary name in /proc/PID/comm.	Standalone stressor that appears with its own binary name in /proc/PID/comm.
ui
util

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

README ¶

xtop

Stop monitoring. Start diagnosing.

What is new in v0.50.0

What is new in v0.49.0

New features

Security hardening (fleet hub)

Stability & correctness

TUI / web / tooling

What is new in v0.48.0

What is new in v0.47.11

What is new in v0.47.2

What's new in v0.47.1

What's new in v0.47.0

What's new in v0.39.1

Table of Contents

The Problem

How xtop Solves It

What xtop Answers in Under 5 Seconds

Features

Root-Cause Analysis Engine

RCA Decision Engine (v0.39.1)

Statistical RCA Intelligence (v0.39.1)

Health Levels

17 Interactive Pages

6 Overview Layouts

eBPF Deep Investigation

eBPF Network Security Intelligence (v0.21.0+)

Predictive Exhaustion Warnings

Anomaly Onset Tracking

Network Intelligence

SMART Disk Health

Incident Recording & Replay

Event Detection

Doctor Mode

Application Diagnostics & RCA

How It Works

Supported Applications

Deep Metrics Collection

Credential Configuration

Active Service Detection

Shell Health Widget

Cron Integration

9 Output Modes

Quick Start

One-liner Install

Build from Source

Run

Installation

Requirements

From .deb Package (Ubuntu 22.04/24.04, Debian)

From .rpm Package (Rocky Linux, RHEL, AlmaLinux, Fedora)

Arch Linux (PKGBUILD)

From Source

Uninstall

Documentation

CLI Reference

Key Bindings

Data Sources

Examples

Configuration

Prometheus Metrics

Alert Payloads

Demo Scenarios

Why Not Just Use...

Architecture

Installed Files

Contributing

Roadmap

License

Documentation ¶

Source Files ¶

Directories ¶