eval

Documentation

Overview

Package eval is the harness's code-based quality evaluation: replay a curated per-task gold set through the live cascade and score with deterministic graders (grounding for extract/summarize, label-match for classify/triage). It is the keystone the prompt/exemplar optimizers (Phases 3-4) replay against.

Index

• func AUGRC(pts []RCPoint) float64
• func Aggregate(outs []Outcome) map[string]Report
• func BootstrapDeltaAURC(incConf, headConf []float64, correct []bool, B int, seed int64) (delta, lo, hi float64)
• func BootstrapDeltaMean(inc, cand []float64, B int, seed int64) (delta, lo, hi float64)
• func DeferralCurve(points []OpPoint) (audc, qnc, peak float64)
• func ECE(pts []RCPoint, nBins int) float64
• func Grade(c Case, res core.Result) bool
• func KFoldOOF[M any](n, k int, seed int64, fit func(train []int) M, score func(m M, i int) float64) []float64
• func RiskCoverage(pts []RCPoint) (coverage, risk []float64, aurc, eaurc float64)
• func SortedTasks(m map[string]Report) []string
• type Case
  • func LoadCases(path string) ([]Case, error)
• type OpPoint
• type Outcome
  • func Run(ctx context.Context, r Runner, cases []Case) []Outcome
• type RCPoint
• type Report
• type Runner

Functions

func AUGRC

func AUGRC(pts []RCPoint) float64

AUGRC is the area under the Generalized Risk-Coverage curve (Traub et al. 2024, "Overcoming Common Flaws in the Evaluation of Selective Classification Systems"). Unlike AURC (selective risk = cumLoss/k), the GENERALIZED risk at coverage rank k is the JOINT risk (cumLoss / N, NOT / k) — it does not blow up at low coverage, so it is more stable. pts are ranked by confidence DESC internally. Returns the mean generalized risk over all coverage levels k=1..N. Lower = better. Returns 0 for empty input.

func Aggregate

func Aggregate(outs []Outcome) map[string]Report

Aggregate rolls outcomes up per task. Correct-defer is a valuable failure: it is NOT counted as an error (the frontier model handles it); only an accepted-wrong result is a true failure.

func BootstrapDeltaAURC

func BootstrapDeltaAURC(incConf, headConf []float64, correct []bool, B int, seed int64) (delta, lo, hi float64)

BootstrapDeltaAURC resamples the N aligned (incConf, headConf, correct) tuples B times with replacement; each resample computes AURC(incumbent) - AURC(head) over the resampled set. Returns the mean delta and the [2.5, 97.5] percentile CI. Deterministic given seed (math/rand with the seed). delta>0 means the head has LOWER AURC (better). The adoption rule is lo > 0 (CI excludes zero). Requires len(incConf)==len(headConf)==len(correct); returns (0,0,0) if N<2 or B<1.

func BootstrapDeltaMean

func BootstrapDeltaMean(inc, cand []float64, B int, seed int64) (delta, lo, hi float64)

BootstrapDeltaMean is a PAIRED bootstrap of the mean difference (candidate − incumbent) over aligned per-item scores — the P6 flywheel's adoption metric for a candidate planner prompt (e.g. goal-reached 0/1 per trajectory). It returns the mean delta and its 95% CI. The SAME fail-closed rule as the AURC gate applies: ADOPT only when lo>0 (candidate strictly better), BLOCK on hi<0, else inconclusive. n<2 or B<1 => zeros (nothing to conclude).

func DeferralCurve

func DeferralCurve(points []OpPoint) (audc, qnc, peak float64)

DeferralCurve computes cascade cost-quality summaries over operating points (Jitkrittum et al. 2024, arXiv:2410.10347; Cost-Aware Contrastive Routing arXiv:2508.12491). Points sorted by cost; cost normalized to [0,1] over the observed range. AUDC = trapezoidal area under quality-vs-normalized-cost (higher = better quality per cost). Peak = max quality. QNC = smallest normalized cost at which peak quality is reached (lower = matches its best quality more cheaply).

func ECE

func ECE(pts []RCPoint, nBins int) float64

ECE computes Expected Calibration Error using nBins equal-width bins over the predicted probability range [0, 1]. For each bin b containing n_b points:

contribution = (n_b / N) * |mean_predicted_b - mean_correct_b|

ECE = sum over non-empty bins. Lower = better calibrated. Returns 0 for empty input. If nBins <= 0 it is clamped to 10. pts need not be sorted.

func Grade

func Grade(c Case, res core.Result) bool

Grade returns whether an ACCEPTED result is correct for the gold case. classify/triage: the model's chosen label/decision must equal Case.Expect. extract/summarize: graded by grounding against the source (no reference needed). Callers must only Grade accepted (res.OK && !res.Deferred) results.

func KFoldOOF

func KFoldOOF[M any](n, k int, seed int64, fit func(train []int) M, score func(m M, i int) float64) []float64

KFoldOOF returns an out-of-fold score for each of n items. Items are shuffled (seeded) then split into k folds; for each fold, fit() is called with the TRAINING indices (all items not in the fold) and returns a model M, then score(M, i) scores each held-out index i. Deterministic given seed. If k<2 it is set to 2; if k>n it is set to n.

func RiskCoverage

func RiskCoverage(pts []RCPoint) (coverage, risk []float64, aurc, eaurc float64)

RiskCoverage computes the discrete risk-coverage curve with AURC and E-AURC. Field-standard selective-prediction metric (Geifman & El-Yaniv 2017; discrete form Ding et al. 2020): sort by confidence DESC; selective risk at coverage k/N = mean 0/1 loss over the k most-confident predictions; AURC = mean selective risk over all k. E-AURC subtracts the oracle AURC (perfect ranking: all correct first), isolating the reducible part. Lower = better.

func SortedTasks

func SortedTasks(m map[string]Report) []string

SortedTasks returns the task keys of a report map in stable order.

Types

type Case

type Case struct {
	Task   string         `json:"task"`
	Input  string         `json:"input"`
	Params map[string]any `json:"params,omitempty"`
	Expect string         `json:"expect,omitempty"` // gold label/decision (classify/triage); optional for extract/summarize
}

Case is one gold-set instance with a defined success condition.

func LoadCases

func LoadCases(path string) ([]Case, error)

LoadCases reads a JSONL gold set; missing file => empty, no error. Empty and malformed (non-JSON / empty-Task) lines are skipped.

type OpPoint

type OpPoint struct {
	Label   string
	Cost    float64 // avg cost per query (e.g. tokens-out)
	Quality float64 // accuracy in [0,1]
}

OpPoint is one cascade operating point: average cost and achieved quality.

type Outcome

type Outcome struct {
	Case        Case
	Accepted    bool
	Correct     bool // among accepted
	Deferred    bool
	TokensOut   int
	Margin      float64
	Escalations int
}

Outcome is the per-case eval result.

func Run

func Run(ctx context.Context, r Runner, cases []Case) []Outcome

Run replays each gold case through r and grades the accepted ones.

type RCPoint

type RCPoint struct {
	Confidence float64
	Correct    bool
}

RCPoint is one (confidence, correctness) observation for selective prediction.

type Report

type Report struct {
	Task             string  `json:"task"`
	N                int     `json:"n"`
	Accepted         int     `json:"accepted"`
	AcceptedCorrect  int     `json:"accepted_correct"`
	Deferred         int     `json:"deferred"`
	AccuracyAccepted float64 `json:"accuracy_accepted"` // correct / accepted
	DeferRate        float64 `json:"defer_rate"`
	TokensOut        int     `json:"tokens_out"`
	AccPer1kTok      float64 `json:"acc_per_1k_tok"` // correct / (tokensOut/1000)
}

Report aggregates outcomes for one task.

type Runner

type Runner interface {
	Run(ctx context.Context, req core.Request) core.Result
}

Runner is satisfied by *pipeline.Pipeline (Run(ctx, req) core.Result).