tinystat

README

tinystat

A Go library and command for evaluating whether two or more sets of measurements are statistically different. It does this by performing a Welch's t-test at a particular confidence level, making it suitable for small sets of measurements (e.g., multiple runs of a benchmark). It's inspired largely by FreeBSD's ministat (written by Poul-Henning Kamp).

Imagine we have the results of different animals' SAT scores. Each animal took the SATs multiple times, and we're assuming that differences between each animal's attempts are measurement error (i.e., normally distributed). We can test for differences as follows:

$ tinystat iguana chameleon leopard

 1.5 k  +
        |
        |
        |
        |
        |
  1000  +                                             |
        |                              |              |
        |                        +-----------+  +-----------+
        |              |         |           |  |           |
        |              |         |           |  +-----*-----+
        |              |         |     *     |  |           |
   500  +              |         +-----------+  +-----------+
        |        +-----------+   |           |        |
        |        |     *     |   +-----------+
        |        +-----------+         |
        |        +-----------+         |
     0  +--------------|-----------------------------------------------
                    iguana         chameleon       leopard

File       N  Mean    Stddev
iguana     7  300.00  238.05  (control)
chameleon  5  540.00  299.08  (no difference, p = 0.050)
leopard    6  643.50  240.09  (643.50 > 300.00 ± 293.97, p = 0.050)

As you can see, despite the superficial differences between the iguana's scores and the chameleon's scores, there is no statistically significant difference between the two at a 95% confidence level. The leopard, on the other hand, has statistically significantly different scores.

Documentation

Overview

Package tinystat provides the ability to compare data sets using Welch's t-test at various levels of confidence.

Index

• type Difference
  • func Compare(control, experiment Summary, confidence float64) Difference
  • func (d Difference) Significant() bool
• type Summary
  • func Summarize(data []float64) Summary
  • func (s *Summary) StdDev() float64
  • func (s *Summary) StdErr() float64

Types

type Difference

type Difference struct {
	Delta            float64 // Delta is the absolute difference between the samples' means.
	CriticalValue    float64 // CriticalValue is the maximum allowed Delta at the given confidence level.
	RelDelta         float64 // RelDelta is the ratio of Delta to the control mean.
	RelCriticalValue float64 // RelCriticalValue is the ratio of CriticalValue to the control mean.
	P                float64 // P is the p-value for the test.
}

Difference represents the statistical difference between two Summary values.

func Compare

func Compare(control, experiment Summary, confidence float64) Difference

Compare returns the statistical difference between the two summaries using a two-tailed Welch's t-test. The confidence level must be in the range (0, 100).

func (Difference) Significant

func (d Difference) Significant() bool

Significant returns true if the difference is statistically significant.

type Summary

type Summary struct {
	N        float64 // N is the number of measurements in the set.
	Mean     float64 // Mean is the arithmetic mean of the measurements.
	Variance float64 // Variance is the sample variance of the data set.
}

A Summary is a statistical summary of a normally distributed data set.

func Summarize

func Summarize(data []float64) Summary

Summarize analyzes the given data set and returns a Summary.

func (*Summary) StdDev

func (s *Summary) StdDev() float64

StdDev returns the standard deviation of the sample.

func (*Summary) StdErr

func (s *Summary) StdErr() float64

StdErr returns the standard error of the sample.