ftdc

README

FTDC-utils

This repository has a Go library and a command-line utility for working with MongoDB Full-Time Diagnostic Data Capture (FTDC) files.

Compatible with FTDC files produced by MongoDB 4.x through 8.x.

The command-line utility, ftdc, helps you:

• decode decode diagnostic files into raw JSON output
• export export each sample as a JSON document suitable for importing into MongoDB
• stats read diagnostic file(s) into aggregated statistical output
• compare compare statistical output

Installation

# for the ftdc command
go install github.com/jenunes/ftdc-utils/cmd/ftdc@latest
# for the Go library, imports as 'ftdc'
go get github.com/jenunes/ftdc-utils

Usage

Decode

Usage:
  ftdc decode [OPTIONS] FILE...

        --start=<TIME>    clip data preceding start time (layout UnixDate)
        --end=<TIME>      clip data after end time (layout UnixDate)
    -m, --merge           merge chunks into one object
    -o, --out=<FILE>      write diagnostic output, in JSON, to given file
    -s, --silent          suppress chunk overview output
    FILE:                 diagnostic file(s)

Export

Usage:
  ftdc export [OPTIONS] FILE...

        --start=<TIME>    clip data preceding start time (layout UnixDate)
        --end=<TIME>      clip data after end time (layout UnixDate)
    -o, --out=<FILE>      write output, in JSON, to given file instead of STDOUT
    -i, --include=<FILE>  include only keys from the given file, one line per key
    -s, --silent          suppress chunk overview output
    FILE:                 diagnostic file(s)

Stats

Usage:
  ftdc stats [OPTIONS] FILE...

        --start=<TIME>    clip data preceding start time (layout UnixDate)
        --end=<TIME>      clip data after end time (layout UnixDate)
    -o, --out=<FILE>      write stats output, in JSON, to given file
    FILE:                 diagnostic file(s)

Compare

Usage:
  ftdc compare [OPTIONS] STAT1 STAT2

    -e, --explicit             show comparison values for all compared metrics;
                               sorted by score, descending
    -t, --threshold=<FLOAT>    threshold of deviation in comparison (default: 0.2)
    STAT1:                     statistical file (JSON)
    STAT2:                     statistical file (JSON)

Documentation

Index

• Variables
• func Chunks(r io.Reader, c chan<- Chunk) error
• type Chunk
  • func (c *Chunk) Clip(start, end time.Time) bool
  • func (c *Chunk) Expand(includeKeys map[string]bool) []map[string]int64
  • func (c *Chunk) Map() map[string]Metric
  • func (c *Chunk) Stats() (s Stats)
• type CmpScore
• type CmpScores
  • func Proximal(a, b Stats) (score float64, scores CmpScores, ok bool)
  • func (s CmpScores) Len() int
  • func (s CmpScores) Less(i, j int) bool
  • func (s CmpScores) Swap(i, j int)
• type DocType
• type Metric
• type MetricStat
• type Stats
  • func ComputeStats(r io.Reader) (cs []Stats, err error)
  • func ComputeStatsInterval(r io.Reader, start, end time.Time) (cs []Stats, err error)
  • func MergeStats(cs ...Stats) (m Stats)

Variables

var CmpThreshold float64 = 0.2

CmpThreshold is the threshold for comparison of metrics used by the Proximal function.

Functions

func Chunks

func Chunks(r io.Reader, c chan<- Chunk) error

Chunks takes an FTDC diagnostic file in the form of an io.Reader, and yields metric chunks (type 1) on the given channel. Type 0 and type 2 documents are silently skipped for backward compatibility. The channel is closed when there are no more chunks.

Types

type Chunk

type Chunk struct {
	Metrics []Metric
	NDeltas int
}

Chunk represents a 'metric chunk' of data in the FTDC (type 1).

func (*Chunk) Clip

func (c *Chunk) Clip(start, end time.Time) bool

Clip trims the chunk to contain as little data as possible while keeping data within the given interval. If the chunk is entirely outside of the range, it is not modified and the return value is false.

func (*Chunk) Expand

func (c *Chunk) Expand(includeKeys map[string]bool) []map[string]int64

Expand accumulates all deltas to give values of diagnostic data for each sample represented by the Chunk. includeKeys specifies which items should be included in the output. If a value of includeKeys is false, it won't be shown even if the value for a parent document is set to true. If includeKeys is nil, data for every key is returned.

func (*Chunk) Map

func (c *Chunk) Map() map[string]Metric

Map converts the chunk to a map representation.

func (*Chunk) Stats

func (c *Chunk) Stats() (s Stats)

Stats produces Stats for the Chunk

type CmpScore

type CmpScore struct {
	// Metric is the name of the metric being compared
	Metric string

	// Score is the value of the score, in the range [0, 1]
	Score float64

	// Error stores an error message if the threshold was not met
	Err error
}

CmpScore holds information for the comparison of a single metric.

type CmpScores

type CmpScores []CmpScore

CmpScores implements sort.Interface for CmpScore slices

func Proximal

func Proximal(a, b Stats) (score float64, scores CmpScores, ok bool)

Proximal computes a measure of deviation between two sets of metric statistics. It computes an aggregated score based on compareMetrics output, and compares it against the CmpThreshold.

Return values: score holds the numeric rating (1.0 = perfect), scores is the sorted list of scores for all compared metrics, and ok is whether the threshold was met.

func (CmpScores) Len

func (s CmpScores) Len() int

func (CmpScores) Less

func (s CmpScores) Less(i, j int) bool

func (CmpScores) Swap

func (s CmpScores) Swap(i, j int)

type DocType

type DocType int32

DocType identifies the type of document in an FTDC file.

const (
	// DocMetadata is a metadata document (type 0), collected on startup/rotation.
	DocMetadata DocType = 0

	// DocMetricChunk is a compressed metric chunk (type 1), the primary data type.
	DocMetricChunk DocType = 1

	// DocPeriodicMetadata is a periodic metadata document (type 2), new in MongoDB ~7.x.
	DocPeriodicMetadata DocType = 2
)

type Metric

type Metric struct {
	// Key is the dot-delimited key of the metric.
	Key string

	// Value is the value of the metric at the beginning of the sample.
	Value int64

	// Deltas is the slice of deltas, which accumulate on Value to yield the
	// specific sample's value.
	Deltas []int64
}

Metric represents an item in a chunk.

type MetricStat

type MetricStat struct {
	// Avg is the mean of the metric's deltas (first derivative).
	Avg int64

	// Var is the variance (related to the absolute second derivative).
	Var int64
}

MetricStat represents basic statistics for a single metric

type Stats

type Stats struct {
	Start    time.Time
	End      time.Time
	Metrics  map[string]MetricStat
	NSamples int
}

Stats represents basic statistics for a set of metric samples.

func ComputeStats

func ComputeStats(r io.Reader) (cs []Stats, err error)

ComputeStats takes an FTDC diagnostic file in the form of an io.Reader, and computes statistics for all metrics on each chunk.

func ComputeStatsInterval

func ComputeStatsInterval(r io.Reader, start, end time.Time) (cs []Stats, err error)

ComputeStatsInterval takes an FTDC diagnostic file in the form of an io.Reader, and computes statistics for all metrics within the given time frame, clipping chunks to fit.

func MergeStats

func MergeStats(cs ...Stats) (m Stats)

MergeStats computes a time-weighted merge of Stats.