README
¶
ptop — process top
ptop is an interactive TUI for deep inspection of processes. Linux is the
rich target (eBPF); macOS runs a reduced "Tier 1" set via libproc + Mach.
Live diagnosis of CPU, syscalls, network, I/O, memory, threads, and file
descriptors — without restarting, instrumenting, or changing a line of code
in the target.
| Tab | Shows | eBPF source |
|---|---|---|
| F1 Overview | CPU sparkline, top syscalls, threads, I/O, FDs, network, heap/leak, events | aggregate |
| F2 Syscalls | per-call frequency, latency, live event stream | raw_syscalls:sys_{enter,exit} |
| F3 Network | TCP/UDP/UNIX connections with state, RTT, Tx/Rx | sock:inet_sock_set_state + sock:sock_{send,recv}_length (kprobes before 6.4) |
| F4 Threads | per-TID state; on-CPU, asleep and run-queue % of each window; lock graph (futex) | sched:sched_switch + futex tracepoints |
| F5 I/O | dual throughput, top files, latency histogram | VFS read/write/fsync syscall tracking |
| F6 FDs | live FD table by type, with bytes and active flag | /proc/<pid>/fd + open/close events |
| F7 Timeline | unified event stream tagged by category | all of the above |
Snapshot
A real go test dump from internal/tui/dump_test.go. Every panel matches
what the live binary renders against a real PID.
⬡ ptop │ api-server PID 1 Go 1.25 RUNNING 15 fds uptime 00:00 │ 18:06:31
F1 Overview │ F2 Syscalls │ F3 Network │ F4 Threads │ F5 I/O │ F6 FD │ F7 Timeline q quit · / filter · p pause
┌──────────────────────────────────────────────────────────────────────────────────┐┌──────────────────────────────────────────────────────┐
│ ▸ CPU ││ ▸ I/O THROUGHPUT │
│ ⡀⡀⡀ ⡀⡀ ⡄ ⡀ ⡀⡀ ⡀⡀⡄ ⡀⡄⡀⡀⡀⡄⡄⡀⡄⡀ ⡀⡄⡀⡀⡀ ⡀⡄⡀ ⡄⡀⡄⡀⡀ ⡀⡄⡀⡀ ⡀⡀ ⡀⡀ 20%││ ⡏⡆⡏⡄⡇ ⡄ ⡟⡟⡏⡄⡟ ⡟⡟⡄⡆⡏ ⡏⡟⡇ ⡆ ⡆⡟⡇⡀⡀⡏⡇⡿⡀⡀⡀⡇ read/s │
│ cpu usage││ 494.2KB │
│ ││⡄⡇⡇⡟⡆ ⡇ ⡆⡀⡏ ⡆⡄⡇⡄⡄⡀⡄⡏⡇⡏⡏⡀⡇⡀⡀⡀⡏⡏⡿⡄⡇⡄⡆⡏⡀ ⡟ write/s│
│ ││ 333.5KB │
└──────────────────────────────────────────────────────────────────────────────────┘└──────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────┐┌──────────────────────────────────────────────────────┐
│ ▸ TOP SYSCALLS ││ ▸ FILE DESCRIPTORS │
│poll ████████████████████████████████████████████████████████████ 195││file ████████████████████████████████████████ 5│
│read ███████████████████████████████████████████████████████████░ 194││socket ████████████████████████████████░░░░░░░░ 4│
│write ████████████████████████████████████████████████████████░░░░ 184││pipe ████████████████████████████████░░░░░░░░ 4│
│openat ███████████████████████████████████████████████████████░░░░░ 181│└──────────────────────────────────────────────────────┘
│fstat ██████████████████████████████████████████████░░░░░░░░░░░░░░ 151│┌──────────────────────────────────────────────────────┐
│getpid ████████████████████████████████████████████░░░░░░░░░░░░░░░░ 143││ ▸ NETWORK │
│epoll_wait ███████████████████████████████████████████░░░░░░░░░░░░░░░░░ 142││TCP → 10.0.1.5:5432 WAIT 42ms │
│recvmsg █████████████████████████████████████████░░░░░░░░░░░░░░░░░░░ 135││TCP ↔ 10.0.0.1:443 ESTABLISHED 8ms │
└──────────────────────────────────────────────────────────────────────────────────┘└──────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────┐┌──────────────────────────────────────────────────────┐
│ ▸ THREADS ││ ▸ EVENT STREAM │
│▶ main ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 34% ││18:06:31.367 CPU preempted after 12ms │
│■ worker-1 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ -- ⏳ mutex-A ││18:06:31.367 SYS futex WAIT mutex-A │
│▶ worker-2 ████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 18% ││18:06:31.367 LCK mutex-A released │
│· gc ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ -- ⏳ nanosleep ││18:06:31.367 I/O write /var/log/app/api.log 512B │
│■ http-pool ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ -- ⏳ epoll_wait ││18:06:31.367 FD openat → fd=15 /tmp/tmpXXXX │
└──────────────────────────────────────────────────────────────────────────────────┘└──────────────────────────────────────────────────────┘
F1-F7 tabs · q quit · p pause · / filter · s snapshot · e export eBPF kernel 6.8 · cpu from sched_switch
A live recording (vhs script in assets/demo.tape) will
replace this section soon.
Requirements
- Linux, kernel 5.8+ (BTF + ring buffer +
CAP_BPF) amd64orarm64- For full mode: root, or the binary with all five capabilities below —
cap_bpf,cap_perfmon,cap_sys_admin,cap_dac_read_search,cap_sys_ptrace+ep - For building from source: Go 1.25+,
clang,libbpf-dev,bpftool
Platform support
macOS has no eBPF and never will — XNU is a different kernel. ptop falls back
to libproc + Mach ("Tier 1") and tells you which tier it started in. Most
panels still carry real data; the ones backed by kernel probes stay empty by
design, and the ? overlay marks them.
| Subsystem | Linux (eBPF) | macOS (libproc + Mach) |
|---|---|---|
| CPU, threads, memory, heap | full | yes |
| File descriptors + events | full | yes |
| I/O throughput + history | full | yes |
| Network connections | full | yes |
| Syscall counts (aggregate) | full | yes |
| Timeline | full | yes |
| Per-syscall trace (F2) | full | no public API |
| Per-file I/O latency (F5) | full | no public API |
| Lock graph (F7) | full | no public API |
| Signals, exec lineage, LSM events | full | no public API |
The three unavailable panels would need DTrace, which SIP restricts, or kdebug, which is undocumented and root-only. See issue #22 for the research.
Install
Homebrew (Linux and macOS):
brew install trentas/tap/ptop
Or grab the pre-built Linux binaries (amd64/arm64) published on each tag. The asset name carries the version, so resolve it first:
VER=$(curl -fsSL https://api.github.com/repos/trentas/ptop/releases/latest \
| sed -n 's/.*"tag_name": *"v\([^"]*\)".*/\1/p')
curl -fsSL "https://github.com/trentas/ptop/releases/download/v$VER/ptop-$VER-linux-$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/').tar.gz" | tar xz
sudo install ptop /usr/local/bin/
Or build from source:
git clone https://github.com/trentas/ptop.git
cd ptop
make # gen + vet + test + build-ebpf
Run
sudo ./bin/ptop --pid <PID> # full mode (eBPF)
./bin/ptop --pid <PID> --no-ebpf # /proc-only, no root
sudo ./bin/ptop --pid <PID> --fps 10 # higher render rate
sudo ./bin/ptop --pid <PID> --export # save JSON snapshot on exit
# Stream events headless (no TUI) over gRPC + JSONL
sudo ./bin/ptop --pid <PID> --serve unix:///run/ptop.sock --export
# Same stream over TCP, encrypted and mutually authenticated (see below)
sudo ./bin/ptop --pid <PID> --serve tcp://10.1.2.3:50051 \
--serve-tls-cert /etc/ptop/tls/tls.crt \
--serve-tls-key /etc/ptop/tls/tls.key \
--serve-tls-client-ca /etc/ptop/tls/ca.crt
# Target a whole container/pod by cgroup instead of a PID (headless only)
sudo ./bin/ptop --cgroup /kubepods.slice/.../cri-containerd-<id>.scope --serve unix:///run/ptop.sock
sudo ./bin/ptop --cgroup <container-id> --serve unix:///run/ptop.sock
# Capture TLS plaintext around libssl (OFF by default; sensitive — see below)
sudo ./bin/ptop --pid <PID> --tls-bytes 256 --serve unix:///run/ptop.sock --export
# Heap probe: drop it entirely, or make it exact instead of sampled (see below)
sudo ./bin/ptop --pid <PID> --disable heap
sudo ./bin/ptop --pid <PID> --heap-sample-bytes 0
sudo ./bin/ptop --pid <PID> --overhead-budget 5 # detach it if it costs the target > 5% for 3 windows
# Name the functions the CPU is going into — sampled, beside the exact ns
sudo ./bin/ptop --pid <PID> --serve unix:///run/ptop.sock # on by default
sudo ./bin/ptop --pid <PID> --cpu-sample-hz 250 # finer, costs more
sudo ./bin/ptop --pid <PID> --disable cpuprof # off
# One fixed window, then a bounded JSON report on stdout — for agents and scripts
sudo ./bin/ptop --pid <PID> --for 10s > report.json # every probe but heap
sudo ./bin/ptop --pid <PID> --for 30s --probes cpu,cpuprof,syscalls --top 20
# Which collectors will run with the privileges you actually have (no PID needed)
./bin/ptop --caps
# Name a STRIPPED third-party binary: symbols by build-id, off-image (see below)
sudo ./bin/ptop --pid <PID> --symbol-cache /srv/symbols # a bundle on disk, no network
sudo ./bin/ptop --pid <PID> --debuginfod # servers from $DEBUGINFOD_URLS
Capture report (
--for, #156): ptop attaches its probes, opens a window of the length asked for (1s to 5m), and prints oneCaptureReport(proto/report.proto) as JSON on stdout. Nothing else goes to stdout. It is what an agent reads instead of writing its own bpftrace: per-axis totals over the window, distributions of what is read reading by reading, and top-N lists (--top, default 10, at most 50) that carry the totals of what they cut. The rules a reader needs are on the message itself; the one that trips people: a total covers its section'sspan_ms, from the axis's first reading in the window to its last, not the whole window. Heap is off unless--probesnames it, and named, it runs under a 5%--overhead-budgetunless you set one.--tlsis refused.--exportbeside it also writes the window's raw events. Exit codes:0the window ran (some probes may not have attached:target.probessays which),3the target exited first (the report covers what was seen),4nothing attached,130interrupted,1/2an error before the window opened or a usage error.
CPU attribution (
--cpu-sample-hz, #125): the CPU axis reports two different things and keeps them apart.cpuis how much — nanoseconds the scheduler actually accounted, exact.cpu_profileis where — the functions a stack sampler caught the target running in, with a sample count and a share. Sampling is the only affordable way to get the second and the wrong way to get the first, so the two are separate payloads and ptop never multiplies one by the other. The sample count is published with every share on purpose: a process at 2.5% of a core draws ~150 samples a minute, which is enough for a rough top-N and not enough to claim a function went from 12% to 18%. The profile also reports the rate the kernel actually delivered next to the one requested — those differ by 10-20% on a quiet host — and how many samples it could not name at all. (That last one is not about frame pointers: a sample's leaf comes from the interrupted registers, not from unwinding, so a target built without them still gets named. What it loses is the depth of the stackResolveStackreturns.)Its sites are user frames only, and the profile says so with
stack_scope: USER(#152). Time the target spends in the kernel, such as a syscall, a page fault or an interrupt taken on its behalf, is credited to the user frame that entered the kernel.kernel_mode_samples, on the profile and on each site, counts the samples that caught the CPU in kernel mode. A function at 40% carrying 35 points of syscall then reads as such instead of as 40% of application code. The kernel functions themselves are not named, because that needs kallsyms.
TLS payload capture (
--tls/--tls-bytes N): uprobes the target's libssl (SSL_write/SSL_read) to record plaintext before encryption / after decryption — handy for debugging your own service's encrypted traffic without a MITM proxy. It is off by default, stream/export-only (no live panel), and the payload bytes (which may contain credentials/PII) are captured only with--tls-bytes N(capped at 4096/call). Go and statically-linked targets have no libssl, so capture is unavailable there.
Keys
| Key | Action |
|---|---|
F1–F7 (or 1–7, Tab/Shift+Tab) |
switch tab |
p, Space |
pause / resume |
/ |
filter (cycles types in F6) |
? |
help overlay (collector status with eBPF/proc/mock source) |
s |
one-shot JSON snapshot |
e |
toggle continuous JSONL export |
q, Ctrl+C |
quit |
Permissions
The recommended setup is to grant capabilities once and run unprivileged:
sudo setcap cap_bpf,cap_perfmon,cap_sys_admin,cap_dac_read_search,cap_sys_ptrace+ep ./bin/ptop
./bin/ptop --pid <PID>
cap_bpf,cap_perfmon — what this README taught until v1.4 — is not the
full set. It loads, and then silently runs about three quarters of the probe
set (#117). The remainder does not announce itself: the capture comes out
well-formed and the dropped axes publish as zeros, and zero is a measurement —
it asserts the process did not allocate. ptop --caps prints exactly which
collectors the privileges you have will run, before you take the capture:
$ ./bin/ptop --caps
...
collectors, with the privileges this process holds
✓ cpu will run
✗ heap WILL NOT COLLECT — CAP_SYS_ADMIN (uprobe PMU in perf_event_open)
~ memory eBPF lane blocked, /proc lane still reports — /proc/self/mem unreadable …
What each capability buys, and what its absence costs:
| Capability | Without it | Why |
|---|---|---|
cap_bpf |
nothing loads | creates maps and loads programs (bpf(2)) |
cap_perfmon |
nothing attaches | opens the perf events tracepoints and kprobes hang off |
cap_sys_admin |
no heap (and no --tls) |
the dynamic perf_uprobe PMU in perf_event_open(2) is gated on it, and CAP_PERFMON does not satisfy that gate. Both heap lanes attach by uprobe, and heap is the only axis carrying func and file:line |
cap_dac_read_search |
most of the set: every tracepoint collector, plus memory, network, heap and --tls |
see below — the setcap above is itself the cause. Two reads it restores: <tracefs>/events/…/id, which every tracepoint attach needs, and /proc/self/mem, which every kprobe- and uprobe-type program's load needs |
cap_sys_ptrace |
nothing at all, for a target owned by another user | pid-mode targeting stats /proc/<pid>/ns/pid to resolve the target's pid namespace, so every eBPF collector fails its filter — not just the uprobe and symbolization paths that read /proc/<pid>/{exe,maps,map_files,root}. All ptrace-mode reads |
The trap in cap_dac_read_search. The kernel marks a process
non-dumpable at exec whenever the new credentials are not a subset of the old
— which is exactly what happens when your shell execs a setcap'd binary — and
a non-dumpable process's own /proc/self/* is reassigned to root. ptop, running as uid 1000, then cannot
read its own /proc/self/mem — which is precisely where cilium/ebpf reads the
running kernel's version out of the vDSO, and every kprobe-type program is
loaded with that version stamped in. ptop's two collectors whose object files
contain a SEC("kprobe/…") — memory and network — therefore fail to load,
before any attach is attempted. So do heap and --tls: a SEC("uprobe/…")
program loads as BPF_PROG_TYPE_KPROBE and takes the same version read, which
is why granting cap_sys_admin without cap_dac_read_search recovers
nothing — heap never reaches the PMU it was missing. /sys/kernel/tracing,
where every tracepoint attach reads its numeric event id, is 0700 root:root
besides. The mechanism recommended so you do not need root is the one that
needs CAP_DAC_READ_SEARCH to undo itself.
Measured on Linux 7.0.0, everything else granted: against a root-owned
target, nine of the twelve collectors fail —
stat pid namespace of <pid>: permission denied for eight of them and
open /proc/<pid>/exe for heap. Only signals and lifecycle survive, and
only because they filter on a global pid of their own. Adding cap_sys_ptrace
recovers all nine. Against a target that is yours, the same four caps
without cap_sys_ptrace attach everything — so omitting it costs nothing if
ptop only ever watches your own processes.
cap_sys_admin is close to root, and in a container sharing the host pid
namespace it is root. Grant it deliberately, and say so to whoever owns the
host. Running without it is a legitimate choice — --caps is then the record
of what you gave up, and --disable heap makes the omission explicit rather
than incidental.
Running as root grants all five and is the simplest thing that works:
sudo ./bin/ptop --pid <PID>
If something is wrong (kernel too old, unprivileged_bpf_disabled=1, missing
caps), ptop prints an actionable error before opening the TUI:
$ ./bin/ptop --pid 1234
error: eBPF not available
Kernel 5.4 detected — ptop requires Linux 5.8+ (BTF + CAP_BPF).
On older kernels, use --no-ebpf (/proc-only mode).
And if it starts with a partial set, it says so on stderr before collecting rather than leaving it to be inferred from the data:
[ptop] ⚠ capability gap: heap, memory, network will not collect.
Missing: CAP_SYS_ADMIN, CAP_DAC_READ_SEARCH
Their axes publish as zeros, which is not the same as the target not doing it.
Collector sources
Each subsystem is fed by one of three sources, picked at startup. The ?
overlay shows which one is active per tab.
| Tab | --no-ebpf (Linux) |
full mode (eBPF) |
|---|---|---|
| F1 Overview | ✅ CPU + Mem + Threads + I/O + FDs | ✅ refined by tracepoints |
| F2 Syscalls | ❌ no /proc source |
✅ raw_syscalls tracepoint |
| F3 Network | ✅ via /proc/<pid>/net/{tcp,udp,unix} |
✅ + per-conn RTT/bytes |
| F4 Threads | ✅ state + CPU% + wchan | ✅ + off-CPU and run-queue % + futex lock graph |
| F5 I/O | ✅ throughput + iowait | ✅ + top files + histogram |
| F6 FDs | ✅ resolved sockets, bytes, active | ✅ same + active socket detection |
| F7 Timeline | partial — only fd events |
✅ all categories |
/proc sources used in --no-ebpf:
cpu_proc.go—/proc/<pid>/statfields 14-15 (utime+stime)threads_proc.go—/proc/<pid>/task/*/stat+wchanmem_proc.go—/proc/<pid>/statm+ page faultsiowait_proc.go—/proc/<pid>/statfield 42 (delayacct_blkio_ticks)io_proc.go—/proc/<pid>/iofds.go+sockets.go—/proc/<pid>/fd, and the target's own socket tables/proc/<pid>/net/{tcp,tcp6,udp,udp6,unix}(its network namespace, not ptop's — #140)proccontext_linux.go—/proc/<pid>/{status,cgroup,ns/*}→ namespace + cgroup + uid/gid (the execution/container context, #60). When the target runs in a container the derived id (docker:…,containerd:…,kubepods:…, …) shows in the header; the full context rides the--serve/--exportstream (a periodicProcContext, plus uid/gid/cgroup_id stamped on every event).
eBPF programs in internal/bpf/programs/:
syscalls.bpf.c— raw_syscalls/sys_{enter,exit}cpu.bpf.c— sched_switch → the target's on-CPU nanosecondsio.bpf.c— read/write + filesystem semantics (denials/deletes/renames), onraw_syscalls(#159)network.bpf.c— sock tracepoints + tcp kprobes + connection errors (RST/retransmit); reads the socket's address through the kernel's BTFthreads.bpf.c— sched_switch → per-thread time on-CPU, asleep and runnable (byprev_state), with the interval still open counted at read time (#150); sched_wakeup ends a sleep where it ends, and every wait for a CPU goes into a log2 histogram (#151)futex.bpf.c— wait/wake → lock graph;signal:signal_deliversettles a wait a signal stopped (#164)memory.bpf.c— mmap/brk/page-fault countersheap.bpf.c— libc malloc/free uprobes → live-heap + lifetime + leak suspectscalib.bpf.c— an empty uprobe: the trap cost, calibrated on the host at startup (#154)goalloc.bpf.c—runtime.mallocgcuprobe → Go allocation sites (rate + volume), stack sampled by bytes allocatedgopause.bpf.c— entry uprobes on the Go runtime's stop and restart of the world → every stop-the-world pause, exact, as a histogram (Go targets only)jvmpause.bpf.c— its twin on a HotSpot JVM (#189): entry uprobes onSafepointSynchronize::begin/endinlibjvm.so→ every safepoint, GC or not; and HotSpot's ownhotspot:gc__begin/gc__endUSDT probes → which of them were collections. Two histograms. Alibjvm.sostripped of.symtab, or built without DTrace probes, loses one lane, and the probe set says which (JVM targets only)
Any subsystem can be switched off with --disable <name,...> — see
what it costs for why you might. Naming
one switches off every lane it has, its /proc reader included, because the
flag is a statement about what is running and a subsystem still reporting
through its cheaper lane would make that statement false. The view for it then
simulates, as it does for any subsystem with no collector behind it, and the
stream's probe set reports it disabled.
signal.bpf.c—signal:signal_generate→ signals delivered, with senderproc.bpf.c—sched_process_{fork,exec,exit}→ exec-lineage subtreesecurity.bpf.c— PROT_EXECmmap/mprotect+ SELinux AVC denialstls.bpf.c— libsslSSL_write/SSL_readuprobes → plaintext (opt-in--tls)
What it costs the process it watches
Measured, not asserted — the harness, methodology and raw data are in
bench/. ptop used to claim overhead <0.5% with nothing behind it;
that claim is gone, and this is what replaced it.
The cost is not a constant. ptop's expensive probe hangs off the allocator, so what it costs depends on how often the target allocates:
| target allocation rate | all probes | without the heap probe | heap probe alone | heap probe unsampled |
|---|---|---|---|---|
| 0 (no allocations) | +0.8% | +0.0% | +1.8% | +0.3% |
| 11k/s | −0.3% | −0.9% | −2.2% | +14.1% |
| 97k/s | +4.8% | −3.7% | +2.3% | +78.6% |
| 1.2M/s | +65.1% | +0.8% | +65.6% | +289.3% |
| 9.6M/s | +824.4% | +31.8% | +739.7% | +1968.8% |
Target CPU time per unit of work, median of 3 runs, on a 10-core aarch64 host
(kernel 7.0). The noise floor there is ±1.8%, measured from the row that
allocates nothing — where every cell should read 0% and does not. Nothing below
that magnitude is resolved, which is why several cells read as zero rather than
as their literal sign. The 97k/s row carries a ±50% spread of its own and
resolves only its last column; the per-cell spreads are in
bench/results/.
Three things follow, all actionable:
Everything except the heap probe is free, at every rate tested — syscalls,
CPU, threads, I/O, network, locks, signals, security and lifecycle together sit
inside the noise floor. The old <0.5% was, by accident, about right for them.
The heap probe samples, and the last two columns are why. A uprobe runs on
the thread that tripped it, so a stack walk per allocation lands in the
target's CPU accounting — and ptop's CPU axis samples the target on-CPU, so
ptop then reported its own probe as the target's CPU (#108). Taking one stack
per 512KB allocated instead of one per allocation turns +14.1% into nothing at
11k/s and +289.3% into +65.6% at 1.2M/s, and drops ptop's own CPU from 22–72%
of a core to under 1%. --heap-sample-bytes 0 is the unsampled column.
The heap probe is still the entire cost above a million allocations a
second, and sampling does not remove the uprobe trap — only the walk. On a
target allocating that hard, --disable heap is the answer, and it keeps
everything else:
ptop --pid <PID> --disable heap
The capture says what it is costing, every second (#154).
- What is sent: a
CaptureCostevent carries the heap probe's estimated CPU charge to the target's threads, as % of one core, and ptop's own CPU beside it. - How it is computed:
(entries + returns) × program_ns + entries × trap_entry_ns + returns × trap_return_ns, published with every term. - Where the terms come from:
- hits: counted in the kernel;
- program time: timed there on one hit in 64;
- trap cost: the entry trap is calibrated on the host when the capture starts. An empty uprobe goes on a function of ptop's own, and calls are timed against calls with nothing attached. A uretprobe's return trap is not calibrated, because a uretprobe on Go code breaks the Go runtime's stack walks. The libc lane's estimate leaves it out, so it is a floor there; the Go lane has no uretprobe.
- Why calibrate: the trap is most of a hit: on the reference arm64 VM, ~485ns of a ~530ns hit. It differs by CPU and kernel by more than a constant could cover: ~2µs on a CI amd64 runner with the same kernel.
- How close it gets: the selftest's
costcase compares the estimate with what an allocation costs a thread with the probe attached, by that thread's own CPU clock. The estimate lands at 0.87–0.89× of the measured cost.
The table above is the expectation before the capture. The estimate is what the capture itself is costing.
--overhead-budget <pct> acts on it, and is off unless you ask for it.
- The unit is the estimate as a share of the target's CPU in the window,
probe time included (
target_share_pcton the estimate, 0–100). It is not quite the table's overhead: a sharesis an overhead ofs/(1−s), so 5% is ~5.3% and 50% is +100%. The overhead has no value once the probe is most of what the target does, and that is the case a budget is for. - When it acts: once the estimate stays above the budget for three windows in a row, ptop detaches the heap probe. Sampling does not remove the uprobe trap, so detaching is the only remedy.
- How it says so:
- Mid-stream: a
probe_changeevent, in the heap's category, carriesheap: disabledand the reason. Without it, a consumer diffing the capture would see the heap sites vanish and read a −100% regression. - Later handshakes: a subscriber that arrives afterwards gets the current probe set.
- Operators: a line on stderr, and one on the TUI's timeline.
- Mid-stream: a
ptop --pid <PID> --serve unix:///run/ptop.sock --overhead-budget 5
Heap: two lanes, picked from the target
heap.bpf.c probes the libc allocator. The Go runtime never calls it —
runtime.mallocgc carves out of per-P caches backed by mmap'd spans — so a Go
process observed on that lane produces an empty call-site axis. Since the
call-site axis is the only part of the heap surface carrying func/file:line,
that means no link from allocation behaviour back to source for an entire
runtime family.
goalloc.bpf.c attaches one uprobe to runtime.mallocgc, the single funnel
every Go heap allocation passes through (newobject, makeslice, growslice
and reflect.unsafe_New all call it; the size-specialised fast paths are
dispatched from inside it). ptop picks the lane from the target's own
executable — a Go image gets the Go lane, including a cgo build with libc
mapped, since that is where a Go program's allocations actually are.
The lanes measure different things, and the snapshot says which:
lane="libc" |
lane="go" |
|
|---|---|---|
| allocation rate + volume, total | exact | exact |
| per-site split | exact | sampled — sample_bytes > 0 |
| completeness of the site list | total_call_sites + omitted_* |
same |
func / file:line per site |
via .symtab + DWARF |
via .gopclntab |
| live bytes, lifetime, leak suspects | yes | no — live_measured=false |
Nothing frees a Go allocation at a point a probe can observe: the GC sweeper
reclaims spans in bulk, asynchronously, naming no object. So the Go lane sets
live_measured=false and leaves those fields at 0 to mean not measured,
never measured, and zero — a consumer diffing two deploys must check the flag
before comparing them.
The Go lane takes a stack every --heap-sample-bytes of allocation on average
(512KB by default, the rate Go's own heap profiler uses) rather than on every
allocation, because that stack walk is charged to the target — see
what it costs. A sampled allocation is
credited with everything allocated since the previous one, so the totals and
rates stay exact; what becomes an estimate is the split BETWEEN sites, and
sample_bytes on the snapshot says so. Allocations at or above Go's 32KB
large-object boundary always take a sample, so they are never estimated away.
The interval is randomized around that average, because allocation patterns are
periodic and a fixed one would cross at the same point of every cycle and hand
one call site all the samples. --heap-sample-bytes 0 records every allocation,
which is exact and costs what the table below says.
The call-site list says how complete it is
top_call_sites carries the largest few sites, not a census. Three fields say
what it left out, on both lanes:
| field | meaning |
|---|---|
total_call_sites |
distinct call sites the snapshot aggregated, before the cut |
omitted_alloc_count / omitted_alloc_bytes / omitted_live_bytes |
volume carried by the sites that did not fit |
When total_call_sites equals the length of the list, the list is a census
and a site's absence from it means the site allocated nothing. Otherwise a site
absent from the list did strictly less than the omitted totals — the bound that
lets a consumer read absence instead of guessing at it.
That distinction is not academic. A consumer diffing two deploys watches a site drop out of the list and cannot otherwise tell "it stopped allocating" from "it stopped being reported"; those are opposite situations, and reading the first where the second holds reports the largest regression such an engine can express over no evidence at all (#109, sunnysystems/witness#69).
The count is in call sites, which is also the unit of the cut. The kernel keys its aggregate by stack id, and one call site is reached through as many stacks as there are paths into the allocator beneath it — a map that grows produces a fresh one per growth path. Stacks reaching the same site are folded together before the ranking, so a function reached many ways no longer holds several of the slots and pushes out a busier one reached a single way.
Symbolization works on stripped release builds (-ldflags="-s -w"): .symtab
is gone but .gopclntab survives, because the runtime needs it for tracebacks.
Symbolization
A captured stack address becomes func (file:line) through whichever of three
sources the module carries, in that order:
| Source | Gives | Present in |
|---|---|---|
.gopclntab |
func + file:line | every Go binary, stripped ones included |
.symtab / .dynsym |
func | C/C++ not built with -s |
| DWARF line program | file:line | anything built with -g |
/tmp/perf-<pid>.map |
func + file:line | JIT runtimes (Node, JVM) |
A module with none of them degrades to module+0xoffset rather than guessing.
Symbolization costs ptop memory in proportion to the target binary, not to how
many collectors use it. Each binary is parsed once and shared, and its line
table and DWARF are mapped from the file instead of copied (#137). What stays
on ptop's heap is the Go function index: about 130 MB for a 394 MB
otelcol-contrib, where a capture peaks at around 260 MB RSS. Size a memory
limit for the largest binary you will attach to.
Symbols that live outside the binary
Those four sources all travel inside the mapped image. The case they leave
unnamed is a third-party binary shipped stripped — built with -s, no DWARF,
no perf map — where the heap axis, the only one carrying func and
file:line, produces addresses and nothing more.
What is missing there is a symbol source, not an identity: every frame already carries the module's GNU build-id, which is the key the ecosystem uses to match a binary with symbols published separately. Two flags turn that key into names (#119):
ptop --pid <PID> --symbol-cache /srv/symbols a bundle on disk, no network
ptop --pid <PID> --debuginfod servers from $DEBUGINFOD_URLS
ptop --pid <PID> --debuginfod-urls https://debuginfod.example named outright
--symbol-cache <dir> is a local symbol store laid out as
<dir>/<build-id>/debuginfo — the same layout elfutils' debuginfod client
uses, so an already-primed client cache works as-is, and so does a vendor
symbol bundle unpacked into it. On its own it reaches no network at all. With
a server configured it is also where a fetched file is written, so a module is
downloaded once per host rather than once per capture; it then defaults to
$DEBUGINFOD_CACHE_PATH, else $XDG_CACHE_HOME/debuginfod_client, else
~/.cache/debuginfod_client.
Four properties are deliberate:
- A build-id that does not match is absence. A file is parsed and its own
build-id compared to the one asked for before a single symbol is taken from
it; anything else is discarded and the frame stays
module+0xoffset. A wrong symbol is worse than no symbol — unlike a bare address it does not look like a gap, and it points a reader at a line of source that has nothing to do with what ran. A download that fails the check is not written to the cache either, so one mis-publishing server cannot poison later runs on the host. - The network is off unless asked for.
$DEBUGINFOD_URLSalone does nothing:--debuginfodis what reads it. A profiler that reaches out to the internet because a binary happened to be stripped is a posture decision, not a convenience. With no flag at all, nothing off-image is consulted — not even a cache directory. - Local resolution stays local. This source is consulted only for what the
mapped image could not answer, and it fills gaps only: a name or a
file:linethe binary supplied is never overwritten, andmodule,offsetandbuild_idalways stay as the mapping reported them. - One lookup per build-id, per process. The hit and the miss are both
remembered, so a stripped module costs one query rather than one per frame.
The lookup is synchronous, so the first frame in such a module waits on it
(bounded by
$DEBUGINFOD_TIMEOUT, default 30s) — the alternative is caching an unresolved frame for the life of the capture.
Symbolization is pid-mode only, here as elsewhere: --cgroup has no single
memory map to resolve against.
JIT'd code has no ELF behind it — V8 and the JVM compile into anonymous
executable memory — so an address in no file-backed mapping is resolved from
the runtime's perf map instead. Node writes one under --perf-basic-prof, the
JVM under perf-map-agent; it is the same side file perf(1) consumes. The
frame's module is reported as [jit], because there is no file to name.
V8 emits the same function once per optimization tier, distinguished only by a
marker: JS:~hotSmall, JS:+hotSmall, JS:^hotSmall. Those are one function
at three addresses. The marker is stripped so they normalize to one identity —
without that, a deploy diff keyed on call-site name would show functions
appearing and vanishing purely because V8 re-tiered them during warm-up.
The map is located through the target's own namespaces: the runtime names the
file with the pid it can see (perf-1.map inside a container), and puts it in
its own /tmp, so ptop reads NSpid from /proc/<pid>/status and crosses
into the target's mount namespace via /proc/<pid>/root.
DWARF sections are copied in while the file is open and parsed on demand, so a
Module still holds no file handle; past a size budget the copy is skipped and
the frame keeps its function name without a line, which is what a
debug-info-heavy C++ image gets instead of hundreds of pinned megabytes.
Event stream (--serve)
The TUI is one consumer of a richer event model. ptop --pid <PID> --serve <addr> runs headless and streams every observation as a typed protobuf Event
over gRPC (package ptop.v1) to any number of unprivileged subscribers — and,
with --export, also to a JSONL file: one protojson SubscribeResponse per
line — the same messages, so a file and a live stream parse identically. ptop
holds CAP_BPF/CAP_PERFMON; subscribers connect with none.
Add --tui to watch the process while it streams: the TUI and the subscribers
become consumers of the same running collectors, so nobody sees half a stream.
Quitting the TUI stops the server.
sudo ./bin/ptop --pid <PID> --serve unix:///run/ptop.sock --tui
Started without --pid, the server has no target of its own: each
subscriber names the process it wants in SubscribeRequest.pid, that pid's
collectors start with its first subscriber and are released when its last one
disconnects. --serve-max-targets (default 8) caps how many run at once, since
each target carries its own eBPF programs.
sudo ./bin/ptop --serve unix:///run/ptop.sock # subscribers pick the target
Transport security
The stream carries process internals — heap call sites, filesystem paths and,
with --tls-bytes, the target's TLS plaintext. What protects it depends on the
endpoint:
| Endpoint | Protection | Flags |
|---|---|---|
unix:///path |
Filesystem: the socket is created 0600 and unlinked on exit |
none (TLS flags are refused here) |
tcp://host:port + TLS |
Encrypted; the subscriber verifies ptop | --serve-tls-cert + --serve-tls-key |
tcp://host:port + mTLS |
Encrypted and only subscribers holding a certificate from your CA are served | the two above + --serve-tls-client-ca |
tcp://host:port cleartext |
None — anyone who reaches the port reads everything | --serve-insecure (explicit opt-in, warns on stderr) |
A tcp:// endpoint with no certificate and no --serve-insecure is refused
at startup: cleartext on the wire is a decision, not a default. Binding all
interfaces (0.0.0.0/::) is refused either way — bind loopback or a specific
interface IP. TLS 1.2 is the floor.
The certificate and the client CA bundle are re-read from disk whenever they change, so a rotated secret (cert-manager and friends) is picked up on the next handshake with no restart. If a rotation lands broken — a key caught half-written — the last good material keeps serving and the reason goes to stderr, because dropping every subscriber is the worse failure.
Not to be confused with
--tls/--tls-bytes, which are the opposite direction: those capture the target's TLS plaintext.--serve-tls-*encrypts ptop's own stream.
Targeting: one PID, or a whole cgroup
--pid names one process. --cgroup names a cgroup subtree — a container
or a whole pod — and needs no PID at all, so you can attach to a workload
chosen by Kubernetes identity rather than by looking up what it happens to be
running as. The spec is a cgroup path (absolute, or the root-relative form
/proc/<pid>/cgroup prints) or a container id, which is resolved by searching
the tree; an ambiguous id is refused rather than guessed. Forks and execs
inside the subtree are covered automatically, because the filter matches the
cgroup, not a process.
Every subscriber is told which mode it is getting, as a TargetInfo handshake
in the first StreamMeta of the stream, before any event. That matters because
the modes aggregate differently: in cgroup mode Event.pid is 0 on aggregate
payloads (there is no single process to name) and any pid inside a payload is a
root-namespace pid, since a subtree can span pid namespaces.
The same handshake carries the probe set: every collector this server tried
to run and what became of it — active (with its source, eBPF or /proc),
disabled, failed, or unsupported, plus why. A probe that is not running
emits nothing, which in the events alone is indistinguishable from a target that
never did the thing, so a consumer comparing two captures has to compare their
probe sets before it compares their numbers. active versus failed is the
distinction that is not cosmetic: a probe someone switched off is a knob they can
turn back on, one that failed to attach is a broken deployment. The server says
the same thing on stderr when it starts:
[ptop] probes: 9 active, 1 failed, 3 disabled, 0 unsupported
[ptop] ⚠ 1 collector(s) asked for but NOT attached — … : syscalls (open tracefs: no such file or directory)
And it names the ptop that produced the stream — ptop { version, commit, build_date } — for the same reason one level up: what a field means changes
between releases (until v1.8.1 a connection's latency_ms was never a
measurement), so two captures are only comparable once you know which side of
such a change each is on. A release carries its tag; a build from a checkout
carries Go's version for it and its commit, +dirty for a modified tree.
In pid mode it also carries what the target is built from, target_build:
the Go version, the main package and module, and every dependency module with
its version, checksum and replace. ptop reads it from the target executable's
own build info, which survives strip. A frame's name does not tell
application code from a dependency's (github.com/acme/svc/internal/db.Get and
github.com/jackc/pgx/v5/pgconn.(*PgConn).receiveMessage look alike). The main
module path does, and two captures' module lists give the dependency diff of a
deploy. Of the build settings only vcs.* is sent. -ldflags lands among the
others, and -X main.apiKey=... is a known way to put a secret into a binary.
The field is absent for a target with no Go build info (C, Rust, JVM), for an
executable ptop cannot read, and in cgroup mode, where one process's build would
not be the subtree's. Absent means unknown.
It also carries the machine and the limits the target runs under (#153).
hostis read once from/procand/sys:- the kernel release and
uname -m; - the CPU as
/proc/cpuinfonames it, the online CPU count and SMT; - the core types of a hybrid CPU;
- whether the host is virtualized, and the DMI vendor and product.
- the kernel release and
limitsis the target cgroup'scpu.max,cpu.weight,memory.maxandcpuset.cpus.effective, verbatim, at subscribe time.
Version N on arm64 against N−1 on x86, on another CPU generation, or under a
lowered cpu.max, moves every CPU number and most latencies without a line of
code changing. Two things change while a capture runs: steal on a VM, and limits
under an in-place pod resize. So a HostSample event carries them every second.
The same sample carries the target cgroup's saturation (#151):
throttling: the CPU bandwidth counters fromcpu.stat(nr_periods,nr_throttled,throttled_usec);pressure: PSI for cpu, memory and io (someandfull, avg10/60/300 and the stalled total).
Utilization says how much CPU was used; saturation says whether more was wanted. A good average with a bad p99 inside a container is usually throttling under cpu.max.
The counters are cumulative, as the kernel keeps them. Take the difference between two samples, and a sample dropped under backpressure loses no time.
A file that cannot be read leaves its part absent: no cpu controller for the cgroup, or PSI off (psi=0, cgroup v1). A measured zero is present with zeros, so the two never look alike.
arm64 has no CPU flag for virtualization, so its verdict there is unknown unless
/sys/hypervisor says otherwise. The hostname, instance id, and DMI serial and
UUID are never read: a comparison needs the hardware class, not which machine it
was.
An empty probe set means the server does not report one — never that every probe ran.
Cgroup mode is --serve only (the TUI's header, thread table and fd list are
all one process's) and needs eBPF, since the filter runs in the kernel. It
starts a deliberately smaller set of collectors — syscalls, I/O, network,
locks, CPU and security — and the omissions are structural, not unfinished:
| Not available in cgroup mode | Why |
|---|---|
| memory, threads | RSS and thread enumeration come from /proc/<pid>/statm and /proc/<pid>/task |
| heap (#53), TLS (#55) | uprobes attach into one process's mapped libc/libssl |
| signals (#58), exec lineage (#60) | they filter on a global pid of their own |
Two collectors that do run lose a pid-shaped detail: I/O reports no file paths
(resolving an fd means reading /proc/<pid>/fd) and security call sites stay
in hex (symbolization reads one process's memory map).
Beyond the seven TUI tabs, the stream carries the full process-behavior surface
(each event tagged by category, with uid/gid/cgroup_id stamped on every
envelope):
| Category | Event | What it captures |
|---|---|---|
MEMORY |
HeapEvent / HeapSnapshot |
allocations by call site (symbolized). libc lane: malloc/free paired → live-heap, lifetime, leak suspects. Go lane: rate + volume, live_measured=false |
NETWORK |
NetErrorEvent |
TCP failures: connection refused, reset, retransmits |
NETWORK |
TLSPayloadEvent |
pre-encryption / post-decryption plaintext (opt-in --tls) |
IO |
FSEvent |
filesystem semantics: permission denials, deletes, renames (real paths) |
SIGNAL |
SignalEvent |
signals delivered to the target, with the sending process |
PROCESS |
ProcContext |
namespace + cgroup + uid/gid (container/execution context) |
THREAD |
RunqLatency |
every wait for a CPU the target's threads finished, by length (log2 µs histogram, cumulative) (#151) |
PROCESS |
CaptureCost |
the heap probe's estimated cost to the target and ptop's own CPU, every second, with the estimate's basis (#154) |
PROCESS |
HostSample |
host steal per second; the target cgroup's limits re-read (#153), its CPU throttling and its PSI for cpu, memory and io (#151) |
PROCESS |
ProcLifecycleEvent |
exec lineage: fork/exec/exit across the descendant subtree |
SECURITY |
SecurityEvent |
runtime PROT_EXEC mappings (JIT/RWX), SELinux LSM denials |
High-rate events reference a captured stack by id; the ResolveStack RPC
symbolizes it on demand (addr → func (file:line), build-id keyed).
Some Event payloads, shown unwrapped for readability — on the wire and in the
JSONL each one arrives inside a SubscribeResponse ({"event":{…}}), which is
also where the target handshake and the drop notices live:
{"tsUnixNano":"…","pid":4242,"category":"CATEGORY_PROCESS","uid":1000,"gid":1000,"cgroupId":"2817","procContext":{"pidNs":"4026532630","cgroup":"/docker/3127f7e31dab…","container":"docker:3127f7e31dab"}}
{"tsUnixNano":"…","pid":4242,"category":"CATEGORY_PROCESS","procLifecycle":{"kind":"exec","pid":4310,"comm":"sh","filename":"/usr/bin/sh"}}
{"tsUnixNano":"…","pid":4242,"category":"CATEGORY_NETWORK","netError":{"kind":"refused","remote":"10.0.0.9:5432"}}
{"tsUnixNano":"…","pid":4242,"category":"CATEGORY_SIGNAL","tid":4242,"signal":{"signal":"SIGPIPE","signo":13,"code":128,"targetTid":4242}}
{"tsUnixNano":"…","pid":4242,"category":"CATEGORY_SECURITY","security":{"kind":"exec-map","op":"mprotect","prot":5,"callSite":{"func":"jit_emit"}}}
The schema lives in proto/event.proto; collectors and
their source-priority selection are shared verbatim between the TUI and the
server, so both see identical data.
Architecture
ptop/
├── cmd/ptop/ entrypoint
├── proto/ event-stream schema (package ptop.v1)
├── internal/
│ ├── bpf/ eBPF programs + loader (build tag `ebpf`)
│ ├── serve/ headless gRPC server (ptop --serve)
│ └── tui/ Bubbletea + Lipgloss views
├── pkg/ importable API surface
│ ├── collector/ /proc + eBPF collectors + shared types
│ ├── streampb/ generated gRPC / protobuf bindings
│ └── symbol/ ELF → symbol resolution (addr → func/file:line)
└── assets/ visual references + vhs script
See CLAUDE.md for the full implementation guide, type
contracts, and conventions.
Development
make # gen + vet + test (both lanes) + build-ebpf — default goal
make gen GOARCH=arm64 # eBPF objects FOR arm64 — build them on an arm64 host
make test # go test -race ./...
make vet # vet in both modes (default + tags=ebpf)
make clean # rm -rf bin/ + the compiled eBPF objects
make lint # golangci-lint (must be installed)
CI runs both lanes (-tags=ebpf and default) on ubuntu-latest. See
CONTRIBUTING.md for the PR flow and commit conventions.
License
MIT. See LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
bench
|
|
|
probecost
command
probecost reports what each loaded eBPF program costs the machine, from the kernel's own accounting rather than from a benchmark delta.
|
probecost reports what each loaded eBPF program costs the machine, from the kernel's own accounting rather than from a benchmark delta. |
|
runner
command
Command runner measures what ptop costs the process it observes.
|
Command runner measures what ptop costs the process it observes. |
|
workload
command
Command workload is the benchmark target: a program whose allocation rate is a knob, so overhead can be measured as a FUNCTION of allocation rate rather than as a single number.
|
Command workload is the benchmark target: a program whose allocation rate is a knob, so overhead can be measured as a FUNCTION of allocation rate rather than as a single number. |
|
cmd
|
|
|
ebpfselftest
command
Stub so `go build ./...` / `go vet ./...` succeed in the default lane — the real self-test only exists in the Linux eBPF build.
|
Stub so `go build ./...` / `go vet ./...` succeed in the default lane — the real self-test only exists in the Linux eBPF build. |
|
ptop
command
|
|
|
internal
|
|
|
report
Package report builds the capture report of `ptop --for` (#156): one fixed window over one target, digested into a CaptureReport an agent can read whole.
|
Package report builds the capture report of `ptop --for` (#156): one fixed window over one target, digested into a CaptureReport an agent can read whole. |
|
serve
Package serve implements ptop's headless streaming mode (ptop --serve): it builds the same collector.Set the TUI uses, fans every collector's output into one stream, and serves it over gRPC to any number of unprivileged subscribers.
|
Package serve implements ptop's headless streaming mode (ptop --serve): it builds the same collector.Set the TUI uses, fans every collector's output into one stream, and serves it over gRPC to any number of unprivileged subscribers. |
|
pkg
|
|
|
streampb
Package streampb holds the generated protobuf bindings for ptop's event stream (proto package ptop.v1, defined in proto/event.proto).
|
Package streampb holds the generated protobuf bindings for ptop's event stream (proto package ptop.v1, defined in proto/event.proto). |
|
symbol
Package symbol resolves runtime stack addresses captured by ptop's eBPF tracers into human-readable frames (function + file:line).
|
Package symbol resolves runtime stack addresses captured by ptop's eBPF tracers into human-readable frames (function + file:line). |