calibration

Documentation

Overview

Package calibration measures how well stated confidences match observed outcomes — the reliability complement to package stats, which owns significance. A well-calibrated confidence of 0.8 is correct about 80% of the time. Pure: values in, values out, no I/O.

The formulas — Expected and Maximum Calibration Error and the Brier score, over equal-width confidence bins comparing each bin's accuracy to its mean stated confidence — are ported from unified-thinking's benchmarks/evaluators/calibration.go, with one fix: every metric divides by the number of in-range samples actually scored, not the raw input length, so an out-of-range sample cannot skew the result.

Index

• type Bucket
• type Report
  • func Compute(samples []Sample) Report
• type Sample

Types

type Bucket

type Bucket struct {
	Count      int     // samples in this bin
	Confidence float64 // mean stated confidence within the bin
	Accuracy   float64 // fraction correct within the bin
}

Bucket is one confidence bin's calibration: how many samples fell in it, the mean confidence stated within it, and the fraction that were correct. A well-calibrated bin has Accuracy close to Confidence.

type Report

type Report struct {
	ECE     float64  // Expected Calibration Error: sample-weighted mean |accuracy − confidence|
	MCE     float64  // Maximum Calibration Error: the worst bin's |accuracy − confidence|
	Brier   float64  // Brier score: mean squared (confidence − outcome)
	Samples int      // in-range samples scored (out-of-range confidences are skipped)
	Buckets []Bucket // non-empty bins, ordered low confidence to high
}

Report is the calibration of a sample set: the three summary errors and the per-bin breakdown they derive from. Lower ECE, MCE, and Brier are better; each lies in [0,1], and MCE is always at least ECE.

func Compute

func Compute(samples []Sample) Report

Compute returns the calibration Report for samples. A sample whose Confidence falls outside [0,1] is skipped and excluded from every metric. Empty or all-skipped input yields the zero Report. Pure.

type Sample

type Sample struct {
	Confidence float64 // stated confidence, in [0,1]
	Correct    bool    // whether the prediction was right
}

Sample is one recorded prediction: the confidence stated when it was made and whether it turned out correct.