README
¶
cohenskappa
Cohen's kappa for Go — inter-rater agreement between two raters. Unweighted, linear, and quadratic weighting, plus custom weight matrices.
Zero dependencies. Verified against scikit-learn.
go get github.com/tobyvee/cohenskappa
Quickstart
import "github.com/tobyvee/cohenskappa"
alice := []string{"spam", "ham", "spam", "spam", "ham"}
bob := []string{"spam", "ham", "ham", "spam", "ham"}
k, err := cohenskappa.Score(alice, bob) // unweighted
Ordinal categories — a 1–5 severity scale, where a 4-vs-5 mix-up should count for less than a 1-vs-5 one:
k, err := cohenskappa.ScoreOrdinal(alice, bob, cohenskappa.Quadratic)
Full control — fix the category order, include categories nobody used, supply your own weights:
s := cohenskappa.Scorer[string]{
Labels: []string{"low", "medium", "high"}, // your order, not the sorted one
Weighting: cohenskappa.Linear,
}
k, err := s.Score(alice, bob)
Already have a confusion matrix? Skip the ratings:
m, _ := cohenskappa.NewMatrix([][]float64{{10, 2, 1}, {3, 8, 2}, {1, 2, 9}})
k, _ := m.Kappa(cohenskappa.Unweighted)
m.ObservedAgreement() // p_o — you usually want to report this too
m.ExpectedAgreement() // p_e
Verified against the exact value of kappa
The formula is three lines. The edge cases are where implementations quietly disagree, and there is usually nothing to catch it. So this package is not tested against its own assumptions — it is tested against ground truth.
Kappa is exactly rational whenever the counts and weights are integers, which
for ratings data they always are. So the true value is computable with no
floating point at all (Python's stdlib fractions.Fraction), and that — not
another float64 implementation — is what the tests assert against.
Measured across the corpus, against the exact value:
| worst absolute error | correctly rounded | |
|---|---|---|
| cohenskappa (Go) | 3.33e-16 | 47.3% |
| scikit-learn 1.9.0 | 4.996e-16 | 45.6% |
Both are good float64 implementations. Neither is the truth — which is precisely why the oracle is worth having, and why agreeing with scikit-learn was never enough on its own.
Parity with scikit-learn is also asserted, because that is what anyone porting from Python is comparing against:
- 1,476 golden fixtures generated from
sklearn.metrics.cohen_kappa_score, covering all three weightings across binary, multiclass (k up to 50), ordinal, imbalanced, degenerate, and large-N (100k) inputs — replayed on everygo test, with no Python required. - CI regenerates them from live scikit-learn and fails on any diff, so a green build proves parity now, not as of whenever someone last ran the generator.
- Differential fuzzing against a live scikit-learn — 50,000 random cases, zero divergence.
- Native Go fuzzing on every public entry point, judged against the exact value
rather than against the statistic's legal range. That distinction is the whole
point: an invariant-only fuzzer asks "is this number legal", cannot tell a
correct 0.83 from a wrong 0.84, and so never sees arithmetic that quietly drifts.
Because kappa is rational, the true value is computable in
math/bigand every one of the ~100 million execs is a check against truth.
Worst observed disagreement with scikit-learn across the corpus: 4.4e-16, about 2 ULP. (Getting there needed compensated summation — a naive sum drifts far enough at k=50 to move the 12th significant figure.)
And because the oracle, the Go code and scikit-learn all descend from the same
reading of the same formula — so a shared misreading would go unnoticed by all
three — the package is also differentially tested against R's irr and
psych, written by other people in another language who never saw our spec. That
is a check on conventions rather than arithmetic: category order, weight
direction, what "undefined" means. Worst deviation over 400 random cases: 6.0e-15
(psych), 2.6e-15 (irr). It also turned up two real defects — in irr, not in
us. See below.
Scoring is allocation-flat in the number of ratings: 768 bytes and 7 allocations whether you score a thousand items or ten million (~34M ratings/sec).
Three things worth knowing
Category order matters for weighted kappa — and only for weighted kappa.
Linear and quadratic weights are computed from the distance between category
indices, so reordering the categories changes the answer. Unweighted kappa is
invariant to it. The API encodes this: Score takes any comparable type and
derives an order freely, because it provably cannot matter; ScoreOrdinal
requires an ordered type and sorts ascending (matching scikit-learn's default);
and a weighted Scorer with no Labels returns ErrNoLabelOrder rather than
silently picking one of several valid answers for you.
A corollary: a category that appears in Labels but in nobody's ratings is not
decoration. It occupies an index and shifts every category after it, changing the
weighted distances that span it. A 5-point scale on which nobody happened to pick
"3" is still a 5-point scale, and this package lets you say so.
Neither of those is a hypothetical. R's irr::kappa2 — a widely used package —
gets both wrong, which is how we found out they were worth designing around:
- It drops unused categories, and you cannot stop it (its first line coerces a
factor data frame to characters, destroying the levels). On a 5-point scale
where nobody picked "3", true quadratic κ is
13/14= 0.9286;irrreturns 0.8649 — the answer for the collapsed 4-point scale — with no warning. - Its default category order is not sorted: it appends any category that only
one rater used to the end of the scale. On one of our random cases that moved
linear κ from
-3/47to0.04. (This one is documented; passsort.levels = TRUE.)
Both are pinned as tests. If you are comparing this package's numbers against R and they differ, these two are the first things to check.
Kappa is undefined when every rating lands in one category. Expected
disagreement is zero, so the statistic is genuinely 0/0. This package returns
NaN and ErrUndefined:
k, err := cohenskappa.Score(allYes, allYes)
if errors.Is(err, cohenskappa.ErrUndefined) {
// Both raters said "yes" to everything. They agreed perfectly — but there is
// no chance-agreement baseline to correct against, so kappa has no value.
}
Returning 0 here — as a naive (p_o − p_e) / (1 − p_e) guard does — would
report "no agreement beyond chance" for two raters who agreed on every single
item. That is wrong in the worst way: it looks right. It also happens constantly
in real labelling data, whenever a batch turns out to be single-class.
scikit-learn agrees, incidentally: it returns NaN, raises
UndefinedMetricWarning, and makes you pass replace_undefined_by if you want a
substitute.
Score, ScoreOrdinal and Matrix.Kappa always return a value in [-1, 1].
That sounds like it should be free, and it isn't. Kappa is 1 − observed/expected,
so when the true value sits exactly on a bound the subtraction cancels its
leading digits and the residue can land on the wrong side. Our fuzzer found a
3-point scale — 112 perfectly ordinary ratings — whose exact quadratic kappa is
exactly -1 and which computed as -1.0000000000000004. scikit-learn, irr and
psych all happen to round the other way on that input and stay inside by luck.
Because the true value is always within the range, snapping an out-of-range result to the nearest bound moves it closer to the truth, never further — so this costs nothing in accuracy and gives you a number you can safely format, compare, or feed to something that assumes the bound.
Matrix.KappaWeights deliberately does not make this promise, because for an
arbitrary caller-supplied weight matrix the bound is not true: an asymmetric matrix
can push kappa to -8. There it returns the value as computed.
Scope
Cohen's kappa, two raters. That is the whole package.
Not included, deliberately: Fleiss' kappa or Krippendorff's alpha (different statistics, not flags on this one), and any "interpretation" of the number — the Landis–Koch substantial agreement bands and their cousins are contested conventions, not mathematics, and do not belong in a numerics package.
Development
make test # hermetic; needs no Python and no R
make lint
make bench
make fuzz # all three targets vs the exact oracle, 60s each
make fuzz-long # the soak: 10 minutes each
make fixtures # regenerate goldens from scikit-learn (needs uv)
make parity # regenerate, fail on drift, then differentially fuzz vs scikit-learn
make parity-r # differentially fuzz vs R's irr and psych (needs R + `make r-deps`)
make parity-all # both
The external references are opt-in on purpose: make test is what everyone who
imports this package inherits, and it must never need a scientific-Python or R
install to pass.
Licence
MIT
Documentation
¶
Overview ¶
Package cohenskappa computes Cohen's kappa, the standard statistic for inter-rater agreement between exactly two raters assigning items to mutually exclusive categories.
It supports unweighted kappa and the two standard weighted variants for ordinal categories (linear and quadratic disagreement weights), plus caller-supplied weight matrices. Results are verified against the reference Python implementation, sklearn.metrics.cohen_kappa_score.
Two rules worth knowing before you start ¶
First, category order matters for weighted kappa and not for unweighted kappa. Linear and quadratic weights are functions of the distance between category indices, so reordering the categories changes the result. Unweighted kappa is invariant to any reordering or relabelling. The API reflects this: Score is unweighted and takes any comparable label type, while ScoreOrdinal requires an ordered type (and sorts ascending) so that the order it uses is never a surprise. For full control, set Labels on a Scorer.
Second, kappa is undefined when expected disagreement is zero, which happens whenever every item falls in a single category. The true value there is 0/0. This package returns NaN and ErrUndefined rather than a plausible-looking zero, because a zero would report "no agreement beyond chance" for what is in fact perfect agreement.
Index ¶
Examples ¶
Constants ¶
const MaxCategories = 10_000
MaxCategories bounds the number of categories k that will be scored.
A confusion matrix is k×k, so the memory cost grows quadratically: k = 10,000 already needs 800 MB. Without a bound, passing ratings that are really identifiers or free text (30,000 distinct values, say) quietly tries to allocate 7.2 GB and dies on the operating system's terms rather than returning an error. Kappa is not a meaningful statistic at that cardinality anyway.
Variables ¶
var ( // ErrEmpty is returned when there is nothing to score: empty rating slices, // or a confusion matrix whose counts sum to zero. ErrEmpty = errors.New("cohenskappa: no ratings provided") // ErrLengthMismatch is returned when the two raters' slices differ in // length. Cohen's kappa is defined over paired ratings of the same items. ErrLengthMismatch = errors.New("cohenskappa: rater slices have different lengths") // ErrUnknownLabel is returned when a rating does not appear in an explicitly // supplied label set. Labels are never silently added: an unexpected label // usually means the caller's label set is wrong, and quietly appending it // would shift category indices and change any weighted result. ErrUnknownLabel = errors.New("cohenskappa: rating not present in the label set") // ErrUndefined is returned when kappa is mathematically undefined because // expected disagreement is zero — which happens whenever all ratings fall in // a single category. The true value is 0/0. // // The accompanying float64 is NaN, not zero. A zero would claim "no // agreement beyond chance" for what is in fact perfect agreement, which is // both wrong and plausible; a NaN stays loud if the error is ignored. ErrUndefined = errors.New("cohenskappa: undefined (expected disagreement is zero)") // ErrBadLabels is returned for a label set this package will not score: // empty, containing duplicates (which would collapse two categories onto one // index), containing a NaN (which can never match itself as a map key), or // holding more than [MaxCategories] categories. // // The message is deliberately generic; the wrapped error says which of those // it was. It used to read "must be non-empty and free of duplicates", which // then produced the self-contradicting "must be non-empty and free of // duplicates: 10001 categories, limit is 10000". ErrBadLabels = errors.New("cohenskappa: invalid label set") // ErrNoLabelOrder is returned when a weighted kappa is requested without a // category order to weight against: a Scorer with a Linear or Quadratic // Weighting (or a custom Weights matrix) but no Labels. // // Weighted kappa is a function of the distance between category indices, so // the answer depends on the category order. Deriving one from the data would // mean silently choosing which of several different, equally valid answers // the caller gets. Set Scorer.Labels, or use ScoreOrdinal, which sorts the // categories ascending and says so. ErrNoLabelOrder = errors.New("cohenskappa: weighted kappa needs an explicit category order (set Scorer.Labels, or use ScoreOrdinal)") // ErrBadWeights is returned for a malformed weight matrix: not square, wrong // size for the label set, negative, non-finite, or non-zero on the diagonal. ErrBadWeights = errors.New("cohenskappa: weight matrix must be square, non-negative, finite, and zero on the diagonal") // ErrBadMatrix is returned for a malformed confusion matrix: not square, // negative, or non-finite. ErrBadMatrix = errors.New("cohenskappa: confusion matrix must be square, non-negative, and finite") // ErrOverflow is returned when the counts (or a custom weight matrix) are so // large that the arithmetic cannot be carried out in float64 without // overflowing to infinity. // // Every individual count can be finite while their sums, or the products of // the marginals, are not. That used to produce a silent NaN — the zero on the // weight diagonal times an infinite expected count is 0*Inf = NaN — which is // precisely the "plausible-looking answer to an impossible question" this // package exists to refuse. ErrOverflow = errors.New("cohenskappa: counts are too large to score in float64 without overflow") )
Sentinel errors returned by this package. Compare with errors.Is; the returned errors wrap these with context (which index, which label, which dimension), so match on the sentinel and print the wrapped error.
Functions ¶
func Score ¶
func Score[T comparable](a, b []T) (float64, error)
Score computes unweighted Cohen's kappa between two raters' labels for the same items. a[i] and b[i] are the two raters' verdicts on item i.
Categories are discovered from the data. Their order does not matter and cannot affect the result: unweighted kappa is invariant to any reordering or relabelling of the categories. That is exactly why Score accepts any comparable type while ScoreOrdinal does not.
It returns ErrUndefined when every rating falls in a single category, where kappa is 0/0.
Example ¶
The common case: two annotators labelling the same items with nominal (unordered) categories.
package main
import (
"fmt"
"github.com/tobyvee/cohenskappa"
)
func main() {
alice := []string{"spam", "ham", "spam", "spam", "ham", "ham", "spam"}
bob := []string{"spam", "ham", "ham", "spam", "ham", "spam", "spam"}
k, err := cohenskappa.Score(alice, bob)
if err != nil {
panic(err)
}
fmt.Printf("kappa = %.4f\n", k)
}
Output: kappa = 0.4167
Example (Undefined) ¶
Kappa is undefined when every rating falls in one category: expected disagreement is zero, so the statistic is 0/0.
This package returns NaN and ErrUndefined rather than a plausible-looking 0. A 0 would report "no agreement beyond chance" for what is, in fact, two raters agreeing on every single item — wrong, and wrong in a way that looks right. Check for it; it happens constantly in real labelling data.
package main
import (
"errors"
"fmt"
"github.com/tobyvee/cohenskappa"
)
func main() {
// Both annotators marked every item "ok". They agreed perfectly.
alice := []string{"ok", "ok", "ok", "ok"}
bob := []string{"ok", "ok", "ok", "ok"}
k, err := cohenskappa.Score(alice, bob)
if errors.Is(err, cohenskappa.ErrUndefined) {
fmt.Println("kappa is undefined here — every rating is in one category,")
fmt.Println("so there is no chance-agreement baseline to correct against.")
fmt.Printf("the returned value is NaN, not 0: %v\n", k)
return
}
fmt.Println(k, err)
}
Output: kappa is undefined here — every rating is in one category, so there is no chance-agreement baseline to correct against. the returned value is NaN, not 0: NaN
func ScoreOrdinal ¶
ScoreOrdinal computes weighted Cohen's kappa over categories sorted ascending, for ordinal data — a 1–5 severity scale, an A–F grade — where being one category apart is a smaller disagreement than being four apart.
The categories are the sorted union of everything appearing in a and b. Sorted order is what scikit-learn's cohen_kappa_score uses by default, so results match it out of the box.
It requires an ordered type precisely because Linear and Quadratic weights are computed from the distance between category indices, so the answer depends on the category order. If your categories are ordinal but do not sort into the right order ("low", "medium", "high" sort to "high", "low", "medium"), or if some category on the scale might be missing from the data, set Labels on a Scorer instead — both of those change the answer.
Example ¶
Ordinal categories — a 1–5 severity scale — where being one step apart is a smaller disagreement than being four steps apart. Quadratic weighting is the usual choice for rating scales.
package main
import (
"fmt"
"github.com/tobyvee/cohenskappa"
)
func main() {
alice := []int{1, 2, 3, 3, 4, 5, 5, 2}
bob := []int{1, 2, 3, 4, 4, 5, 4, 3}
unweighted, err := cohenskappa.Score(alice, bob)
if err != nil {
panic(err)
}
quadratic, err := cohenskappa.ScoreOrdinal(alice, bob, cohenskappa.Quadratic)
if err != nil {
panic(err)
}
// Unweighted counts a 1-vs-5 mix-up the same as a 3-vs-4 one, so it scores
// these near-miss raters harshly. Quadratic knows they were nearly right.
fmt.Printf("unweighted = %.4f\n", unweighted)
fmt.Printf("quadratic = %.4f\n", quadratic)
}
Output: unweighted = 0.5385 quadratic = 0.8868
Types ¶
type Matrix ¶
type Matrix struct {
// contains filtered or unexported fields
}
Matrix is a confusion matrix over a fixed, ordered set of k categories: Counts[i][j] is the number of items the first rater assigned to category i and the second rater assigned to category j.
Counts are float64 rather than int so that fractional counts (from per-sample weights) fit without an API change. Integer counts below 2^53 are exact in a float64, so nothing is lost by using one.
A Matrix is immutable once constructed. Row and column sums are computed once, at construction.
func NewMatrix ¶
NewMatrix builds a confusion matrix from a square, non-negative, finite k×k slice of counts. The input is copied; later mutation of counts does not affect the returned Matrix.
A category with no observations at all (an all-zero row and column) is valid, not an error. It still occupies an index, and under linear or quadratic weighting it therefore shifts the indices of every category after it — which is exactly the point of including it.
An all-zero matrix is also valid at construction; it is [Kappa] that reports ErrEmpty for it.
Example ¶
If you already have a confusion matrix, skip the ratings entirely. Matrix also gives you observed and expected agreement, which you usually want to report alongside kappa.
package main
import (
"fmt"
"github.com/tobyvee/cohenskappa"
)
func main() {
// Rows are the first rater, columns the second.
m, err := cohenskappa.NewMatrix([][]float64{
{10, 2, 1},
{3, 8, 2},
{1, 2, 9},
})
if err != nil {
panic(err)
}
k, err := m.Kappa(cohenskappa.Unweighted)
if err != nil {
panic(err)
}
fmt.Printf("observed agreement = %.4f\n", m.ObservedAgreement())
fmt.Printf("expected by chance = %.4f\n", m.ExpectedAgreement())
fmt.Printf("kappa = %.4f\n", k)
}
Output: observed agreement = 0.7105 expected by chance = 0.3338 kappa = 0.5655
func (*Matrix) Counts ¶
Counts returns a copy of the confusion matrix. Mutating it does not affect m.
func (*Matrix) ExpectedAgreement ¶
ExpectedAgreement returns p_e, the proportion of items the raters would be expected to agree on by chance alone, given their marginals:
p_e = sum_i (rows[i]/N) * (cols[i]/N)
It is NaN when the matrix is empty.
func (*Matrix) Kappa ¶
Kappa returns Cohen's kappa for the matrix under the given weighting.
The result is always in [-1, 1]. That bound holds mathematically for all three built-in weightings, so a float64 result outside it can only be rounding, and is snapped back to the nearest bound — which is strictly closer to the true value than leaving it out of range. (It is reachable: a 3×3 anti-diagonal whose exact quadratic kappa is exactly -1 otherwise computes as -1.0000000000000004.) Matrix.KappaWeights makes no such promise; see there.
It returns NaN and ErrUndefined when expected disagreement is zero — that is, when every observation sits in a single category — because kappa is 0/0 there. It returns ErrEmpty when the matrix has no observations at all.
func (*Matrix) KappaWeights ¶
KappaWeights returns Cohen's kappa under a caller-supplied k×k disagreement weight matrix, where k is the number of categories. Use it for weighting schemes that Weighting does not cover. The matrix must be square, non-negative, finite, and zero on the diagonal.
Scaling the whole matrix by a positive constant does not change the result: kappa is a ratio of two sums that are each linear in the weights.
Unlike Matrix.Kappa, the result is NOT guaranteed to lie in [-1, 1]. The familiar bound is a property of the built-in weightings, not of the statistic, and an asymmetric weight matrix can push kappa well below -1 (-8.09 on a 2×2, in testing). The value is therefore returned as computed, unclamped: with arbitrary weights there is no theorem to say a number outside the range is an error rather than the answer.
Expected counts follow the convention E[i][j] = rows[i]*cols[j]/N, where rows are the first rater's marginals. This is observable only for an *asymmetric* weight matrix — for any symmetric one (including all of Unweighted, Linear and Quadratic) the transposed convention gives an identical sum. Note that scikit-learn builds its expected matrix the other way round (np.outer of the column sums with the row sums); it has no custom-weight path, so the two never actually disagree, but anyone porting an asymmetric scheme from R's irr or psych should check which convention their source used.
func (*Matrix) ObservedAgreement ¶
ObservedAgreement returns p_o, the proportion of items the two raters put in the same category. It is NaN when the matrix is empty.
type Scorer ¶
type Scorer[T comparable] struct { // Labels fixes the category set and its order. It may contain categories // that appear in neither rater's ratings; those still occupy an index, and // so shift the indices of every category after them. Under [Linear] or // [Quadratic] weighting that changes the answer, which is the point: a // 5-point scale where nobody happened to pick "3" is still a 5-point scale, // and "2" and "4" are still two steps apart on it. // // A rating not present in Labels is an error, never a new category. // // If Labels is nil, categories are discovered from the data in first-seen // order. That is safe for unweighted kappa (the order cannot affect it) and // is refused for weighted kappa (it would silently pick one of several // answers) — see [ErrNoLabelOrder]. Labels []T // Weighting selects a built-in disagreement weighting. Ignored when Weights // is set. The zero value is [Unweighted]. Weighting Weighting // Weights is a k×k disagreement weight matrix, k == len(Labels), overriding // Weighting. See [Matrix.KappaWeights] for the constraints and for the // expected-count convention that matters if it is asymmetric. Weights [][]float64 }
Scorer computes kappa with an explicit category set and weighting. Use it when Score and ScoreOrdinal do not give you enough control: to fix the category order, to include categories that never appear in the data, or to supply your own weight matrix.
The zero Scorer computes unweighted kappa with categories discovered from the data — identical to Score.
Example ¶
Scorer with an explicit label set. Use it when the category order is not the sorted order, or when a category on your scale never appears in the data.
The unused category is not decoration: it occupies an index, so it shifts every category after it and changes the weighted distances that span it. A 5-point scale on which nobody happened to pick "3" is still a 5-point scale.
package main
import (
"fmt"
"github.com/tobyvee/cohenskappa"
)
func main() {
// Nobody rated anything "3", but "3" is still on the scale.
alice := []string{"1", "2", "4", "5", "1", "4"}
bob := []string{"1", "2", "5", "5", "2", "4"}
scale := cohenskappa.Scorer[string]{
Labels: []string{"1", "2", "3", "4", "5"},
Weighting: cohenskappa.Quadratic,
}
withScale, err := scale.Score(alice, bob)
if err != nil {
panic(err)
}
// Drop the unused "3" and 2-vs-4 becomes an adjacent disagreement rather
// than a two-step one. Different question, different answer.
collapsed := cohenskappa.Scorer[string]{
Labels: []string{"1", "2", "4", "5"},
Weighting: cohenskappa.Quadratic,
}
withoutScale, err := collapsed.Score(alice, bob)
if err != nil {
panic(err)
}
fmt.Printf("with the full 5-point scale = %.4f\n", withScale)
fmt.Printf("with the gap collapsed = %.4f\n", withoutScale)
}
Output: with the full 5-point scale = 0.9341 with the gap collapsed = 0.8696
type Weighting ¶
type Weighting int
Weighting selects how much each kind of disagreement counts against the raters.
Unweighted treats every disagreement as equally bad and is the right choice for nominal (unordered) categories. Linear and Quadratic are for ordinal categories — a 1–5 severity scale, say — where confusing "1" with "5" should cost more than confusing "1" with "2". Quadratic punishes distant disagreements hardest and is the usual choice for rating scales.
Linear and Quadratic depend on the order of the category set; Unweighted does not. See the package documentation.
Source Files