Documentation
¶
Overview ¶
Package diag captures a runtime profile bundle from a running flynn process.
Where observe carries agent semantics (what a run did, how much it cost), diag carries process runtime facts: where CPU went, what the heap held, which goroutines were alive, how many file descriptors and child processes the process owned over time. The two are deliberately separate: an operator reading a stuck run needs the second kind of evidence, and it must be capturable from a binary that is already built and already misbehaving.
A bundle is a directory of pprof profiles, a JSONL timeline, and a manifest that names and hashes every member, so a bundle can be moved off the machine and still be shown intact.
A bundle can also watch itself. With Config.Leak set, the timeline sampler fits a growth slope across every counter it records and writes a labelled goroutine profile, a heap profile, and the offending window into the bundle at the moment growth starts, unattended. See leak.go.
Nothing here runs unless a bundle is explicitly started: there is no init, no background goroutine, and no allocation on the disabled path. Start with an empty Config returns a nil *Bundle, and every method on a nil *Bundle is a no-op, so the caller never branches on whether profiling is on.
The package depends only on the standard library, the clock port (so the timeline sampler is driven by an injected clock and stays testable), the secret port (so an argv recorded in the manifest is redacted by the same redactor the rest of the agent uses), and the observe port (so a leak the watchdog finds is reported through the same logger as everything else).
Index ¶
- Constants
- func DefaultThresholds() map[string]Threshold
- func LabelGoroutine(ctx context.Context, kv ...string) context.Context
- func Labeled(ctx context.Context, fn func(context.Context), kv ...string)
- func Profiling() bool
- func RedactArgs(args []string) []string
- type Bundle
- type Config
- type Counter
- type Finding
- type LeakConfig
- type Manifest
- type Member
- type Sample
- type Threshold
Constants ¶
const ( // EnvDir names the bundle directory, equivalent to --profile. EnvDir = "FLYNN_PROFILE" // EnvContention enables the block and mutex profiles, equivalent to // --profile-contention. Any value Go's strconv.ParseBool accepts as true enables it. EnvContention = "FLYNN_PROFILE_CONTENTION" )
Environment variables that turn a bundle on without changing the command line, so a hosted or containerized instance can be profiled in place.
const ( MemberCPU = "cpu.pprof" MemberHeap = "heap.pprof" MemberAllocs = "allocs.pprof" MemberGoroutine = "goroutine.pprof" MemberGoroutineTxt = "goroutine.txt" MemberGoroutineLbl = "goroutine.labels.txt" MemberThreadcreate = "threadcreate.pprof" MemberBlock = "block.pprof" MemberMutex = "mutex.pprof" MemberTimeline = "runtime.jsonl" MemberManifest = "manifest.json" )
Bundle member file names. A reader (see the `flynn diagnose` command) resolves members by these names, so they are part of the bundle's contract.
const ( CounterGoroutines = "goroutines" CounterHeapLive = "heap_live_bytes" CounterOpenFDs = "open_fds" CounterChildProcs = "child_procs" )
The counters the watchdog knows how to read off a Sample. Each names a distinct leak class, because one profile sees only one of them: a run that never closes a file, or never reaps a sandboxed command, shows a flat heap and a flat goroutine count right up to the moment it fails.
const DefaultInterval = time.Second
DefaultInterval is how often the timeline sampler records a line when Config leaves Interval unset. One second is fine enough to fit a growth slope over a run of any interesting length and coarse enough that the sampler's own cost stays in the noise.
const DefaultWindow = 12
DefaultWindow is how many consecutive samples the detector fits before it will fire. At DefaultInterval that is twelve seconds of growth that never once reverses, which no goal loop the agent runs produces and every real leak does.
const EnvLeakWatch = "FLYNN_LEAK_WATCH"
EnvLeakWatch enables the watchdog on a process whose command line cannot be changed, equivalent to --leak-watch. Any value Go's strconv.ParseBool accepts as true enables it. It has no effect without a bundle directory, because the watchdog samples the bundle's timeline and dumps into the bundle.
const Unknown = -1
Unknown is the value a counter takes when it cannot be reported: a platform that does not expose it, or an application that supplied no way to read it. It is negative so that no reader can average, sum, or threshold it into a false zero.
Variables ¶
This section is empty.
Functions ¶
func DefaultThresholds ¶
DefaultThresholds are the floors for the built-in counters, chosen so that a multi-turn run with fan-out does not fire and a real leak does.
The goroutine and fd floors sit above the transient population of a fan-out (a child agent, its bus subscribers, and the descriptors they hold, all of which return to baseline when the child is reaped). The heap floor is a rise of 64 MiB that never once reverses across twelve garbage collections, which retained garbage produces and a working set does not. The child-process floor is low because nothing in the agent legitimately holds eight unreaped children.
func LabelGoroutine ¶
LabelGoroutine attaches kv to the calling goroutine for the rest of its life, and returns a context carrying the same labels so goroutines it starts inherit them. Call it as the first statement of a long-lived goroutine: a subscription pump, a queue worker, a resync loop. Unlike Labeled there is nothing to unwind, because the goroutine's whole lifetime is the scope.
When no bundle is open it returns ctx unchanged and does nothing.
func Labeled ¶
Labeled runs fn with kv attached as pprof labels to the calling goroutine and, by inheritance, to every goroutine fn starts. The labels are removed when fn returns.
This is what turns a wall of identical stacks into an attribution: a goroutine profile of a leaking agent shows several thousand goroutines parked in a channel receive, and nothing in the stack says which action left them there. Labelled, the same profile reads as 1,900 goroutines under one action, and the leak names itself.
When no bundle is open, fn is called directly: no label map, no context value, no allocation. kv is a flat key, value, key, value list; a trailing key with no value is dropped rather than fatal, because a mislabelled profile is a smaller problem than a killed run.
func Profiling ¶
func Profiling() bool
Profiling reports whether a bundle is currently open in this process. Call it before building label values that cost anything to produce; the labelling helpers below check it themselves, so a caller with labels already in hand does not need to.
func RedactArgs ¶
RedactArgs returns args with every credential and every piece of free user text replaced by secret.Redacted. The result is safe to write into a bundle manifest that an operator will attach to a bug report.
It redacts three things: the value of any flag whose name looks like a credential, every argument following a subcommand that takes free text (a goal's objective, an `auth set` key), and any bare argument carrying a recognizable credential prefix. The program path, flag names, subcommands, and ordinary values are preserved, because a manifest whose command line is entirely redacted tells a reader nothing about what was being profiled.
Redaction reuses secret.Text rather than formatting a placeholder here, so the agent has exactly one rendering of a withheld value.
A subcommand is recognized by name wherever it appears among the bare arguments rather than by position. flag's own parse rules cannot be reproduced here (this package does not know which of flynn's flags take a value), and guessing wrong about position would silently leave an objective in the clear.
Types ¶
type Bundle ¶
type Bundle struct {
// contains filtered or unexported fields
}
Bundle is an open profile capture. A nil *Bundle is the disabled bundle: every method on it is a no-op returning nil, so a caller holds one unconditionally.
A Bundle is written by Stop, which is not safe to call concurrently with itself; call it once, from the same goroutine that owns the command's exit path.
func Start ¶
Start opens the bundle directory and begins capture. It returns a nil Bundle and a nil error when cfg.Dir is empty, which is the disabled path: no directory is touched, no goroutine starts, and nothing is allocated.
The caller stops the bundle exactly once, normally from a defer on the process's single exit path. A bundle whose process calls os.Exit is never written, because os.Exit runs no defers; that is a property of os.Exit, not of this package.
func (*Bundle) Annotate ¶
Annotate records a key/value pair in the manifest, so a caller can correlate a bundle with the run it captured once the run's identity is known. It is safe on a nil Bundle and safe for concurrent use.
func (*Bundle) Stop ¶
Stop ends capture, writes the exit-time profiles and the manifest, and closes every member. It is a no-op on a nil Bundle, and a second call is a no-op.
Stop reports the first error it hit but always attempts every remaining member, so a bundle is as complete as the failure allows rather than truncated at it.
type Config ¶
type Config struct {
// Dir is the bundle directory, created if absent. An empty Dir disables capture.
Dir string
// Contention enables the block and mutex profiles. Both make the runtime record
// every blocking event and every contended lock handoff, which costs real time in
// a hot process, so they are off unless asked for.
Contention bool
// Interval is the timeline sampler's period. Zero means DefaultInterval. A
// negative Interval disables the sampler, leaving the rest of the bundle intact.
Interval time.Duration
// Args is the command line to record in the manifest, redacted. Callers pass
// os.Args. An empty Args records no command line.
Args []string
// Clock is the time source for the timeline and the manifest's start and end
// stamps. Nil means clock.System. A test supplies a clock.Manual to drive the
// sampler deterministically.
Clock clock.Timing
// Leak turns the leak watchdog on. Nil is the disabled watchdog. The watchdog
// rides the timeline sampler, so it needs one: a Config with a Leak and a
// negative Interval is an error rather than a silently inert watchdog.
Leak *LeakConfig
// Counters are application-supplied gauges sampled into every timeline line
// alongside the built-in ones, and watched by the watchdog when Leak's thresholds
// name them. This is where a counter this package cannot know about (unremoved
// temporary directories, the event log's size on disk) is registered.
Counters []Counter
// Children reports how many child processes the application has started and not yet
// reaped, and fills the timeline's child_procs. It is read on the sampler goroutine
// once per interval and must not block.
//
// This package cannot answer the question itself. It could walk /proc or take a
// Windows process snapshot, but that costs an open and a read per process on the
// machine on every sample of a bundle meant to be safe to leave on for days, and it
// answers a different question: which processes currently name this pid as their
// parent, a set that pid reuse can populate with strangers. Only the spawners know
// what they spawned, so they are asked. See the procs package.
//
// Nil records Unknown rather than zero. An application that never said what it
// spawned has not thereby proved it spawned nothing, and Unknown restarts the
// watchdog's window instead of feeding it a flat line of false zeroes.
Children func() int
// contains filtered or unexported fields
}
Config describes a bundle to capture. The zero Config is disabled: Start returns a nil Bundle and touches nothing.
type Counter ¶
Counter is an application-supplied gauge sampled alongside the built-in ones and recorded in the timeline under its name. It is how counters this package cannot know about are watched: the number of temporary directories created and not removed, the size of the event log on disk, any other quantity whose steady state is flat.
Read runs on the sampler goroutine, once per interval, and must not block: it delays the timeline while it runs. It returns Unknown for a value it cannot measure this sample, which the detector treats as a gap rather than as a zero.
type Finding ¶
type Finding struct {
// Counter is the counter that fired.
Counter string
// Slope is the fitted growth, in counter units per sample.
Slope float64
// Delta is the rise from the window's first sample to its last.
Delta float64
// First and Last are the window's bounding values.
First, Last float64
// Window is the samples the detector fitted, oldest first.
Window []Sample
// Dumps names the files written for this finding, relative to the bundle
// directory. It is empty when the dump failed, which the log record reports.
Dumps []string
}
Finding is one firing: a counter that grew, by how much, over which samples.
type LeakConfig ¶
type LeakConfig struct {
// Window is how many samples the detector fits. Zero means DefaultWindow. A
// window shorter than 4 is rejected: three points admit any slope.
Window int
// Repeat lets a counter fire more than once in a process. By default a counter
// fires once, because the second dump of a leak that is still leaking says what
// the first already said, and an unattended process must not fill a disk with
// evidence of a fact already recorded.
Repeat bool
// Thresholds maps counter name to the floors that counter must clear. Nil means
// DefaultThresholds. A non-nil map is used as given, so a caller that names only
// CounterGoroutines watches only goroutines: a counter with no threshold is
// recorded in the timeline and never fires.
Thresholds map[string]Threshold
// Logger receives a Warn record on every firing. Nil means observe.NopLogger, in
// which case the dump on disk is the only report.
Logger observe.Logger
}
LeakConfig turns the watchdog on. A nil *LeakConfig on Config is the disabled watchdog: nothing is fitted, nothing is dumped, and the sampler behaves exactly as it does without it.
type Manifest ¶
type Manifest struct {
// BundleID uniquely names this capture.
BundleID string `json:"bundle_id"`
// FlynnVersion is the binary's version string, revision included when it has one.
FlynnVersion string `json:"flynn_version"`
// Revision is the VCS revision the binary was built from, when it is known.
Revision string `json:"revision,omitempty"`
// GoVersion is the Go toolchain the binary was built with. A profile is read
// against the runtime that produced it, so this is not decoration.
GoVersion string `json:"go_version"`
// OS and Arch are the platform the capture ran on.
OS string `json:"os"`
Arch string `json:"arch"`
// NumCPU is the parallelism the scheduler had, needed to read a CPU profile's
// sample counts as a fraction of available time.
NumCPU int `json:"num_cpu"`
// Args is the command line, redacted: an objective and an API key both reach it.
Args []string `json:"args,omitempty"`
// Contention reports whether the block and mutex profiles were captured.
Contention bool `json:"contention"`
// SampleIntervalMs is the timeline sampler's period in milliseconds.
SampleIntervalMs int64 `json:"sample_interval_ms"`
// StartedAt and EndedAt bound the capture window.
StartedAt time.Time `json:"started_at"`
EndedAt time.Time `json:"ended_at"`
// Annotations correlate the bundle with whatever the caller knew about the run,
// such as its id, once that identity existed.
Annotations map[string]string `json:"annotations,omitempty"`
// Members names and hashes every other file in the bundle.
Members []Member `json:"members"`
}
Manifest describes a bundle: what produced it, over what window, and what it contains. It is the first member a reader opens, and the only one that can establish that the others arrived intact.
type Member ¶
type Member struct {
// Name is the member's file name, relative to the bundle directory.
Name string `json:"name"`
// Bytes is the member's size on disk.
Bytes int64 `json:"bytes"`
// SHA256 is the member's content digest, lowercase hex.
SHA256 string `json:"sha256"`
}
Member is one file in a bundle, with the size and digest it had when the manifest was written.
type Sample ¶
type Sample struct {
// T is when the sample was taken, from the injected clock.
T time.Time `json:"t"`
// Goroutines is the live goroutine count. A monotone rise is the clearest leak
// signal the runtime offers.
Goroutines int `json:"goroutines"`
// Threads is the number of OS threads the runtime has ever created. It only
// grows, so a jump means blocking syscalls or cgo, not garbage.
Threads int `json:"threads"`
// HeapAllocBytes is live heap memory; HeapObjects is the count of live objects.
HeapAllocBytes uint64 `json:"heap_alloc_bytes"`
HeapObjects uint64 `json:"heap_objects"`
// HeapLiveBytes is the heap the last garbage collection found reachable, or
// Unknown where the runtime does not report it. HeapAllocBytes read mid-cycle
// counts garbage that has not been collected yet, so it rises and falls with the
// collector; this one moves only when retention moves, which is why the leak
// watchdog fits its slope and not HeapAllocBytes'.
HeapLiveBytes int64 `json:"heap_live_bytes"`
// HeapSysBytes is heap memory obtained from the OS: the ceiling the process has
// actually reached, which is what an operator's memory limit sees.
HeapSysBytes uint64 `json:"heap_sys_bytes"`
// Mallocs and Frees are cumulative object counts. Their difference is HeapObjects,
// and their rate is allocation pressure the CPU profile alone will not show.
Mallocs uint64 `json:"mallocs"`
Frees uint64 `json:"frees"`
// NumGC and GCPauseTotalNs are cumulative garbage-collector work.
NumGC uint32 `json:"num_gc"`
GCPauseTotalNs uint64 `json:"gc_pause_total_ns"`
// OpenFDs is the process's open file descriptor (or handle) count, or -1 where
// the platform does not expose it. A rise here outlives any Go-level leak: it
// ends in "too many open files".
OpenFDs int `json:"open_fds"`
// ChildProcs is how many child processes the application has started and not yet
// reaped, as reported by Config.Children, or -1 when no Children was supplied. The
// agent spawns sandboxed commands; one that is never reaped shows up here and
// nowhere else.
ChildProcs int `json:"child_procs"`
// Extra carries the application-supplied counters from Config.Counters, keyed by
// counter name. It is absent from a sample taken with no such counters, and a
// counter that could not be measured this sample carries Unknown.
Extra map[string]float64 `json:"extra,omitempty"`
}
Sample is one line of the timeline: the process's runtime shape at a moment. A single sample says little; the series is what matters, because a leak is a slope and not a value. D3's watchdog fits that slope live, and `flynn diagnose` diffs two bundles' series after the fact.
Counters the platform cannot report arrive as -1 rather than 0, so a reader never mistakes "not measurable here" for "none".
type Threshold ¶
type Threshold struct {
// MinSlope is the least least-squares slope, in counter units per sample, that
// counts as growth.
MinSlope float64
// MinDelta is the least rise from the window's first sample to its last, in
// counter units, that counts as growth.
MinDelta float64
}
Threshold is what a counter must do, across a full window, to be called a leak. Both floors must be cleared, and they answer different objections: MinSlope rejects growth too slow to matter over the life of a process, and MinDelta rejects growth too small to matter at all.
Source Files