Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrSamplesEqual = errors.New("all equal")
	ErrSampleSize   = errors.New("too few samples")
	ErrZeroVariance = errors.New("zero variance")
)

    Errors returned by DeltaTest.

    Functions

    func ByDelta

    func ByDelta(t *Table, i, j int) bool

      ByDelta sorts tables by the Delta column, reversing the order when larger is better (for "speed" results).

      func ByName

      func ByName(t *Table, i, j int) bool

        ByName sorts tables by the Benchmark name column

        func FormatCSV

        func FormatCSV(w io.Writer, tables []*Table, norange bool)

          FormatCSV appends a CSV formatting of the tables to w. norange suppresses the range columns.

          func FormatHTML

          func FormatHTML(buf *bytes.Buffer, tables []*Table)

            FormatHTML appends an HTML formatting of the tables to buf.

            func FormatText

            func FormatText(w io.Writer, tables []*Table)

              FormatText appends a fixed-width text formatting of the tables to w.

              func NoDeltaTest

              func NoDeltaTest(old, new *Metrics) (pval float64, err error)

                NoDeltaTest applies no delta test; it returns -1, nil.

                func Sort

                func Sort(t *Table, order Order)

                  Sort sorts a Table t (in place) by the given order.

                  func TTest

                  func TTest(old, new *Metrics) (pval float64, err error)

                    TTest is a DeltaTest using the two-sample Welch t-test.

                    func UTest

                    func UTest(old, new *Metrics) (pval float64, err error)

                      UTest is a DeltaTest using the Mann-Whitney U test.

                      Types

                      type Collection

                      type Collection struct {
                      	// Configs, Groups, and Units give the set of configs,
                      	// groups, and units from the keys in Stats in an order
                      	// meant to match the order the benchmarks were read in.
                      	Configs, Groups, Units []string
                      
                      	// Benchmarks gives the set of benchmarks from the keys in
                      	// Stats by group in an order meant to match the order
                      	// benchmarks were read in.
                      	Benchmarks map[string][]string
                      
                      	// Metrics holds the accumulated metrics for each key.
                      	Metrics map[Key]*Metrics
                      
                      	// DeltaTest is the test to use to decide if a change is significant.
                      	// If nil, it defaults to UTest.
                      	DeltaTest DeltaTest
                      
                      	// Alpha is the p-value cutoff to report a change as significant.
                      	// If zero, it defaults to 0.05.
                      	Alpha float64
                      
                      	// AddGeoMean specifies whether to add a line to the table
                      	// showing the geometric mean of all the benchmark results.
                      	AddGeoMean bool
                      
                      	// SplitBy specifies the labels to split results by.
                      	// By default, results will only be split by full name.
                      	SplitBy []string
                      
                      	// Order specifies the row display order for this table.
                      	// If Order is nil, the table rows are printed in order of
                      	// first appearance in the input.
                      	Order Order
                      }

                        A Collection is a collection of benchmark results.

                        func (*Collection) AddConfig

                        func (c *Collection) AddConfig(config string, data []byte)

                          AddConfig adds the benchmark results in the formatted data to the named configuration. If the input is large, AddFile should be preferred, since it avoids the need to read a copy of the entire raw input into memory.

                          func (*Collection) AddFile

                          func (c *Collection) AddFile(config string, r io.Reader) error

                            AddFile adds the benchmark results in the formatted data (read from the reader r) to the named configuration.

                            func (*Collection) AddResults

                            func (c *Collection) AddResults(config string, results []*benchfmt.Result)

                              AddResults adds the benchmark results to the named configuration.

                              func (*Collection) Tables

                              func (c *Collection) Tables() []*Table

                                Tables returns tables comparing the benchmarks in the collection.

                                type DeltaTest

                                type DeltaTest func(old, new *Metrics) (float64, error)

                                  A DeltaTest compares the old and new metrics and returns the expected probability that they are drawn from the same distribution.

                                  If a probability cannot be computed, the DeltaTest returns an error explaining why. Common errors include ErrSamplesEqual (all samples are equal), ErrSampleSize (there aren't enough samples), and ErrZeroVariance (the sample has zero variance).

                                  As a special case, the missing test NoDeltaTest returns -1, nil.

                                  type Key

                                  type Key struct {
                                  	Config, Group, Benchmark, Unit string
                                  }

                                    A Key identifies one metric (e.g., "ns/op", "B/op") from one benchmark (function name sans "Benchmark" prefix) and optional group in one configuration (input file name).

                                    type Metrics

                                    type Metrics struct {
                                    	Unit    string    // unit being measured
                                    	Values  []float64 // measured values
                                    	RValues []float64 // Values with outliers removed
                                    	Min     float64   // min of RValues
                                    	Mean    float64   // mean of RValues
                                    	Max     float64   // max of RValues
                                    }

                                      A Metrics holds the measurements of a single metric (for example, ns/op or MB/s) for all runs of a particular benchmark.

                                      func (*Metrics) Format

                                      func (m *Metrics) Format(scaler Scaler) string

                                        Format returns a textual formatting of "Mean ±Diff" using scaler.

                                        func (*Metrics) FormatDiff

                                        func (m *Metrics) FormatDiff() string

                                          FormatDiff computes and formats the percent variation of max and min compared to mean. If b.Mean or b.Max is zero, FormatDiff returns an empty string.

                                          func (*Metrics) FormatMean

                                          func (m *Metrics) FormatMean(scaler Scaler) string

                                            FormatMean formats m.Mean using scaler.

                                            type Order

                                            type Order func(t *Table, i, j int) bool

                                              An Order defines a sort order for a table. It reports whether t.Rows[i] should appear before t.Rows[j].

                                              func Reverse

                                              func Reverse(order Order) Order

                                                Reverse returns the reverse of the given order.

                                                type Row

                                                type Row struct {
                                                	Benchmark string     // benchmark name
                                                	Group     string     // group name
                                                	Scaler    Scaler     // formatter for stats means
                                                	Metrics   []*Metrics // columns of statistics
                                                	PctDelta  float64    // unformatted percent change
                                                	Delta     string     // formatted percent change
                                                	Note      string     // additional information
                                                	Change    int        // +1 better, -1 worse, 0 unchanged
                                                }

                                                  A Row is a table row for display in the benchstat output.

                                                  type Scaler

                                                  type Scaler func(float64) string

                                                    A Scaler is a function that scales and formats a measurement. All measurements within a given table row are formatted using the same scaler, so that the units are consistent across the row.

                                                    func NewScaler

                                                    func NewScaler(val float64, unit string) Scaler

                                                      NewScaler returns a Scaler appropriate for formatting the measurement val, which has the given unit.

                                                      type Table

                                                      type Table struct {
                                                      	Metric      string
                                                      	OldNewDelta bool // is this an old-new-delta table?
                                                      	Configs     []string
                                                      	Groups      []string
                                                      	Rows        []*Row
                                                      }

                                                        A Table is a table for display in the benchstat output.