mlmetrics

README

ML metrics

Common metrics for evaluation of machine learning models.

Goals:

• Fast!
• Thread-safe
• Support for online evaluation

Supported Metrics

Classification:

• Accuracy
• Confusion Matrix
• F1 Score
• Kappa
• Matthews
• LogLoss
• Precision
• Sensitivity

Regression:

• Mean Absolute Error
• Mean Squared Error
• Root Mean Squared Error
• Mean Squared Error
• R²

Documentation

Documentation and example are available via godoc at http://godoc.org/github.com/bsm/mlmetrics

Example

package main

import (
	"github.com/bsm/mlmetrics"
)

func main() {
	yTrue := []int{2, 0, 2, 2, 0, 1}
	yPred := []int{0, 0, 2, 2, 0, 2}

	mat := mlmetrics.NewConfusionMatrix()
	for i := range yTrue {
		mat.Observe(yTrue[i], yPred[i])
	}

	// print matrix
	for i := 0; i < mat.Order(); i++ {
		fmt.Println(mat.Row(i))
	}

	// print metrics
	fmt.Println()
	fmt.Printf("accuracy : %.3f\n", mat.Accuracy())
	fmt.Printf("kappa    : %.3f\n", mat.Kappa())
	fmt.Printf("matthews : %.3f\n", mat.Matthews())

}

Documentation

Index

• type Accuracy
  • func NewAccuracy() *Accuracy
  • func (m *Accuracy) CorrectWeight() float64
  • func (m *Accuracy) Observe(actual, predicted int)
  • func (m *Accuracy) ObserveWeight(actual, predicted int, weight float64)
  • func (m *Accuracy) Rate() float64
  • func (m *Accuracy) Reset()
  • func (m *Accuracy) TotalWeight() float64
• type ConfusionMatrix
  • func NewConfusionMatrix() *ConfusionMatrix
  • func (m *ConfusionMatrix) Accuracy() float64
  • func (m *ConfusionMatrix) Column(x int) []float64
  • func (m *ConfusionMatrix) F1(x int) float64
  • func (m *ConfusionMatrix) Kappa() float64
  • func (m *ConfusionMatrix) Matthews() float64
  • func (m *ConfusionMatrix) Observe(actual, predicted int)
  • func (m *ConfusionMatrix) ObserveWeight(actual, predicted int, weight float64)
  • func (m *ConfusionMatrix) Order() int
  • func (m *ConfusionMatrix) Precision(x int) float64
  • func (m *ConfusionMatrix) Reset()
  • func (m *ConfusionMatrix) Row(x int) []float64
  • func (m *ConfusionMatrix) Sensitivity(x int) float64
  • func (m *ConfusionMatrix) TotalWeight() float64
• type LogLoss
  • func NewLogLoss() *LogLoss
  • func NewLogLossWithEpsilon(epsilon float64) *LogLoss
  • func (m *LogLoss) Observe(prob float64)
  • func (m *LogLoss) ObserveWeight(prob float64, weight float64)
  • func (m *LogLoss) Reset()
  • func (m *LogLoss) Score() float64
• type Regression
  • func NewRegression() *Regression
  • func (m *Regression) MAE() float64
  • func (m *Regression) MSE() float64
  • func (m *Regression) MSLE() float64
  • func (m *Regression) MaxError() float64
  • func (m *Regression) Mean() float64
  • func (m *Regression) Observe(actual, predicted float64)
  • func (m *Regression) ObserveWeight(actual, predicted, weight float64)
  • func (m *Regression) R2() float64
  • func (m *Regression) RMSE() float64
  • func (m *Regression) RMSLE() float64
  • func (m *Regression) Reset()
  • func (m *Regression) TotalWeight() float64

Examples

• ConfusionMatrix
• LogLoss
• Regression

Types

type Accuracy

type Accuracy struct {
	// contains filtered or unexported fields
}

Accuracy is a basic classification metric. It measures how often the classifier makes the correct prediction. It is the ratio between the weight of correct predictions and the total weight of predictions.

func NewAccuracy

func NewAccuracy() *Accuracy

NewAccuracy inits a new metric.

func (*Accuracy) CorrectWeight

func (m *Accuracy) CorrectWeight() float64

CorrectWeight returns the weight of correct observations.

func (*Accuracy) Observe

func (m *Accuracy) Observe(actual, predicted int)

Observe records an observation of the actual vs the predicted category.

func (*Accuracy) ObserveWeight

func (m *Accuracy) ObserveWeight(actual, predicted int, weight float64)

ObserveWeight records an observation of the actual vs the predicted category with a given weight.

func (*Accuracy) Rate

func (m *Accuracy) Rate() float64

Rate returns the rate of correct predictions.

func (*Accuracy) Reset

func (m *Accuracy) Reset()

Reset resets state.

func (*Accuracy) TotalWeight

func (m *Accuracy) TotalWeight() float64

TotalWeight returns the total weight observed.

type ConfusionMatrix

type ConfusionMatrix struct {
	// contains filtered or unexported fields
}

ConfusionMatrix can be used to visualize the performance of a binary classifier.

Example

yTrue := []int{2, 0, 2, 2, 0, 1}
yPred := []int{0, 0, 2, 2, 0, 2}

mat := mlmetrics.NewConfusionMatrix()
for i := range yTrue {
	mat.Observe(yTrue[i], yPred[i])
}

// print matrix
for i := 0; i < mat.Order(); i++ {
	fmt.Println(mat.Row(i))
}

// print metrics
fmt.Println()
fmt.Printf("accuracy : %.3f\n", mat.Accuracy())
fmt.Printf("kappa    : %.3f\n", mat.Kappa())
fmt.Printf("matthews : %.3f\n", mat.Matthews())

Output:

[2 0 0]
[0 0 1]
[1 0 2]

accuracy : 0.667
kappa    : 0.429
matthews : 0.452

func NewConfusionMatrix

func NewConfusionMatrix() *ConfusionMatrix

NewConfusionMatrix inits a new ConfusionMatrix.

func (*ConfusionMatrix) Accuracy

func (m *ConfusionMatrix) Accuracy() float64

Accuracy returns the overall accuracy rate.

func (*ConfusionMatrix) Column

func (m *ConfusionMatrix) Column(x int) []float64

Column returns the distribution of actual weights for category x.

func (*ConfusionMatrix) F1

func (m *ConfusionMatrix) F1(x int) float64

F1 calculates the F1 score for category x, the harmonic mean of precision and sensitivity.

func (*ConfusionMatrix) Kappa

func (m *ConfusionMatrix) Kappa() float64

Kappa represents the Cohen's Kappa, a statistic which measures inter-rater agreement for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation, as κ takes into account the possibility of the agreement occurring by chance. https://en.wikipedia.org/wiki/Cohen%27s_kappa

func (*ConfusionMatrix) Matthews

func (m *ConfusionMatrix) Matthews() float64

Matthews is a correlation coefficient used as a measure of the quality of binary and multiclass classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient. [source: Wikipedia]

func (*ConfusionMatrix) Observe

func (m *ConfusionMatrix) Observe(actual, predicted int)

Observe records an observation of the actual vs the predicted category.

func (*ConfusionMatrix) ObserveWeight

func (m *ConfusionMatrix) ObserveWeight(actual, predicted int, weight float64)

ObserveWeight records an observation of the actual vs the predicted category with a given weight.

func (*ConfusionMatrix) Order

func (m *ConfusionMatrix) Order() int

Order returns the matrix order (number or rows/cols).

func (*ConfusionMatrix) Precision

func (m *ConfusionMatrix) Precision(x int) float64

Precision calculates the positive predictive value for category x.

func (*ConfusionMatrix) Reset

func (m *ConfusionMatrix) Reset()

Reset resets the state.

func (*ConfusionMatrix) Row

func (m *ConfusionMatrix) Row(x int) []float64

Row returns the distribution of predicted weights for category x.

func (*ConfusionMatrix) Sensitivity

func (m *ConfusionMatrix) Sensitivity(x int) float64

Sensitivity calculates the recall (aka 'hit rate') for category x.

func (*ConfusionMatrix) TotalWeight

func (m *ConfusionMatrix) TotalWeight() float64

TotalWeight returns the total weight observed (sum of the matrix).

type LogLoss

type LogLoss struct {
	// contains filtered or unexported fields
}

LogLoss, aka logistic loss or cross-entropy loss.

Example

// assuming the following three predictions
predictions := []map[string]float64{
	{"dog": 0.2, "cat": 0.5, "fish": 0.3},
	{"dog": 0.8, "cat": 0.1, "fish": 0.1},
	{"dog": 0.6, "cat": 0.1, "fish": 0.4},
}

// create metric, feed with probabilities of actual observations
metric := mlmetrics.NewLogLoss()
for i, actual := range []string{"cat", "dog", "fish"} {
	probability := predictions[i][actual]
	metric.Observe(probability)
}

// print score
fmt.Printf("log-loss : %.3f\n", metric.Score())

Output:

log-loss : 0.611

func NewLogLoss

func NewLogLoss() *LogLoss

NewLogLoss inits a log-loss metric.

func NewLogLossWithEpsilon

func NewLogLossWithEpsilon(epsilon float64) *LogLoss

NewLogLossWithEpsilon inits a log-loss metric with epsilon, a small increment to add to avoid taking a log of zero. Default: 1e-15.

func (*LogLoss) Observe

func (m *LogLoss) Observe(prob float64)

Observe records the predicted probability of the actually observed value. Assuming the predictions were:

[dog: 0.2, cat: 0.5, fish: 0.3]
[dog: 0.8, cat: 0.1, fish: 0.1]
[dog: 0.6, cat: 0.1, fish: 0.4]

And the actual observations were:

• cat
• dog
• fish

Then the recorded values should be:

m.Observe(0.5)
m.Observe(0.8)
m.Observe(0.4)

func (*LogLoss) ObserveWeight

func (m *LogLoss) ObserveWeight(prob float64, weight float64)

ObserveWeight records an observation with a given weight.

func (*LogLoss) Reset

func (m *LogLoss) Reset()

Reset resets state.

func (*LogLoss) Score

func (m *LogLoss) Score() float64

Score calculates the logarithmic loss.

type Regression

type Regression struct {
	// contains filtered or unexported fields
}

Regression is a basic regression evaluator

Example

package main

import (
	"fmt"
	"math"

	. "github.com/bsm/ginkgo"
	. "github.com/bsm/gomega"
	"github.com/bsm/mlmetrics"
)

var _ = Describe("Regression", func() {
	var subject *mlmetrics.Regression

	BeforeEach(func() {
		subject = mlmetrics.NewRegression()
		subject.Observe(26, 25)
		subject.Observe(20, 25)
		subject.Observe(24, 22)
		subject.Observe(21, 23)
		subject.Observe(23, 24)
		subject.Observe(25, 29)
		subject.Observe(27, 28)
		subject.ObserveWeight(28, 26, 2.0)
		subject.Observe(29, 30)
		subject.Observe(22, 18)
	})

	It("should calculate basic stats", func() {
		Expect(subject.TotalWeight()).To(Equal(11.0))
		Expect(subject.MaxError()).To(Equal(5.0))
		Expect(subject.Mean()).To(BeNumerically("~", 24.818, 0.001))
	})

	It("should mean errors", func() {
		Expect(subject.MAE()).To(BeNumerically("~", 2.273, 0.001))
		Expect(subject.MSE()).To(BeNumerically("~", 7.000, 0.001))
		Expect(subject.RMSE()).To(BeNumerically("~", 2.646, 0.001))
		Expect(subject.MSLE()).To(BeNumerically("~", 0.012, 0.001))
		Expect(subject.RMSLE()).To(BeNumerically("~", 0.110, 0.001))
	})

	It("should calculate R²", func() {
		Expect(subject.R2()).To(BeNumerically("~", 0.390, 0.01))

		subject.ObserveWeight(28, 28, 2.0)
		Expect(subject.R2()).To(BeNumerically("~", 0.477, 0.001))
	})

	It("should handle blanks", func() {
		subject.Reset()
		Expect(subject.TotalWeight()).To(Equal(0.0))
		Expect(subject.MaxError()).To(Equal(0.0))
		Expect(subject.Mean()).To(Equal(0.0))
		Expect(subject.MAE()).To(Equal(0.0))
		Expect(subject.MSE()).To(Equal(0.0))
		Expect(subject.RMSE()).To(Equal(0.0))
		Expect(subject.MSLE()).To(Equal(0.0))
		Expect(subject.RMSLE()).To(Equal(0.0))
	})

	It("should handle negative values", func() {
		subject.Observe(-28, -27)

		Expect(subject.MaxError()).To(Equal(5.0))
		Expect(subject.Mean()).To(BeNumerically("~", 20.417, 0.001))
		Expect(subject.MAE()).To(BeNumerically("~", 2.167, 0.001))
		Expect(subject.MSE()).To(BeNumerically("~", 6.500, 0.001))
		Expect(subject.RMSE()).To(BeNumerically("~", 2.550, 0.001))
		Expect(math.IsNaN(subject.MSLE())).To(BeTrue())
		Expect(math.IsNaN(subject.RMSLE())).To(BeTrue())
	})
})

func main() {
	yTrue := []float64{26, 20, 24, 21, 23, 25, 27}
	yPred := []float64{25, 25, 22, 23, 24, 29, 28}

	metric := mlmetrics.NewRegression()
	for i := range yTrue {
		metric.Observe(yTrue[i], yPred[i])
	}

	// print score
	fmt.Printf("count : %.0f\n", metric.TotalWeight())
	fmt.Printf("mae   : %.3f\n", metric.MAE())
	fmt.Printf("mse   : %.3f\n", metric.MSE())
	fmt.Printf("rmse  : %.3f\n", metric.RMSE())
	fmt.Printf("msle  : %.3f\n", metric.MSLE())
	fmt.Printf("rmsle : %.3f\n", metric.RMSLE())
	fmt.Printf("r2    : %.3f\n", metric.R2())

}

Output:

count : 7
mae   : 2.286
mse   : 7.429
rmse  : 2.726
msle  : 0.012
rmsle : 0.110
r2    : 0.162

func NewRegression

func NewRegression() *Regression

NewRegression inits a new metric.

func (*Regression) MAE

func (m *Regression) MAE() float64

MAE calculates the mean absolute error.

func (*Regression) MSE

func (m *Regression) MSE() float64

MSE calculates the mean squared error.

func (*Regression) MSLE

func (m *Regression) MSLE() float64

MSLE calculates the mean squared logarithmic error loss.

func (*Regression) MaxError

func (m *Regression) MaxError() float64

MaxError returns the maximum observed error delta.

func (*Regression) Mean

func (m *Regression) Mean() float64

Mean returns the mean actual value observed.

func (*Regression) Observe

func (m *Regression) Observe(actual, predicted float64)

Observe records an observation of the actual vs the predicted value.

func (*Regression) ObserveWeight

func (m *Regression) ObserveWeight(actual, predicted, weight float64)

ObserveWeight records an observation of the actual vs the predicted value with a given weight.

func (*Regression) R2

func (m *Regression) R2() float64

R2 calculates the R² coefficient of determination.

func (*Regression) RMSE

func (m *Regression) RMSE() float64

RMSE calculates the root mean squared error.

func (*Regression) RMSLE

func (m *Regression) RMSLE() float64

RMSLE calculates the root mean squared logarithmic error loss.

func (*Regression) Reset

func (m *Regression) Reset()

Reset resets state.

func (*Regression) TotalWeight

func (m *Regression) TotalWeight() float64

TotalWeight returns the total weight observed.