Documentation
¶
Overview ¶
Package confhead implements a per-task logistic correctness head. It reads the JSONL ledger, fits a binary logistic regression (pure Go, gradient descent, L2 regularisation) per task over ALL ledger features and metadata signals, and predicts p(correct) for an entry at inference time. Unlike the router (which predicts accepted-at-E2B), confhead predicts whether the model's answer was actually correct — it will be validated (Task 2) and, only if it lowers AURC, wired into the pipeline (Task 4).
Index ¶
- func FeatureRow(e ledger.Entry) map[string]float64
- func Label(e ledger.Entry) (y float64, ok bool)
- func LoadThresholds(path string) map[string]float64
- func SelectThreshold(scores []float64, correct []bool, targetErr float64) float64
- func Train(ledgerPath, labelsPath, outPath string) (string, error)
- type Model
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FeatureRow ¶
FeatureRow builds a feature map for one ledger entry. It copies the 6 Feat fields (absent keys default to 0 after standardise), then appends the 4 metadata signals: margin, truncated, retries, loginput. EXPORTED: Task 2 (out-of-fold validation) reuses this function directly.
func Label ¶
Label derives the binary correctness label for a ledger entry. ok=true only when at least one of Grounded or EscalatedAgreed is non-nil (i.e., a human or Opus ground-truth signal is available). y=1 if the answer was correct (grounded true OR escalation agreed); else 0. EXPORTED: Task 2 uses this to build the validation set.
func LoadThresholds ¶
LoadThresholds reads the per-task conformal p(correct) escalation thresholds written by `confhead-calibrate` (Task 3) at path — a flat JSON object like {"summarize":0.69,"extract":0}. A missing, empty, or unparseable path yields an empty map. The result is always safe to index (a nil map reads as 0), so the pipeline never panics when the head/threshold file is absent.
func SelectThreshold ¶
SelectThreshold computes the per-task escalation threshold tau on p(correct): at runtime the pipeline escalates a call to the larger tier when its p(correct) < tau, keeping (accepting) only the calls with score >= tau.
It chooses the SMALLEST tau such that, among the ACCEPTED rows (score >= tau), the conformal finite-sample error bound is <= targetErr. Smallest qualifying tau means maximal acceptance / minimal escalation while still meeting the per-task error target.
Conformal safety convention (mirrors internal/calibration exactly): the accepted error is bounded by the CRC-adjusted rate
adjusted = (nAccepted * empiricalErr + 1) / (nAccepted + 1)
= (nErr + 1) / (nAccepted + 1)
i.e. the same +1/(n+1) finite-sample slack used by the margin calibration, so the two thresholding paths are consistent. An empty accepted set scores 1.0 (worst case) by the same convention. A PURE-correct accepted set (nErr == 0) is provably below any positive bound, so it scores adjusted error 0 — letting it meet a target of exactly 0 (the perfectly-ranked, zero-target case).
Degenerate cases:
- If accepting EVERYTHING already meets the target (the RAW overall error <= targetErr), no escalation is needed => tau = 0. The short-circuit uses raw (not conformally-adjusted) overall error so a target >= the base error rate maps to tau = 0 exactly, matching the calibration intent.
- If no non-empty accepted set can meet the target — or the inputs are empty or mismatched — tau is set to the maximum observed score, so almost everything escalates. tau is capped at 1.0 so it never exceeds the p(correct) range (a score of exactly 1.0 would then accept nothing, escalating all rows).
scores and correct must be the same length; out-of-fold scores should be used by the caller so tau is not optimistic.
func Train ¶
Train reads the JSONL ledger at ledgerPath plus the correctness-label sidecar at labelsPath, merges them, calls Fit, writes the model to outPath (confhead-weights.json), and returns a human-readable per-task report.
The sidecar (written by the pipeline via ledger.AppendLabel) carries cascade-agreement labels for classify/triage — tasks grounding can't label — so the head can cover them once escalation traffic accrues. An empty labelsPath (or a missing file) behaves like "no sidecar". The report notes how many rows per task came from the sidecar.
Types ¶
type Model ¶
type Model struct {
// contains filtered or unexported fields
}
Model holds per-task logistic-regression weights loaded from disk (or Fit). A nil *Model is safe to call — Predict returns -1 (sentinel: no head).
func Fit ¶
Fit trains one logistic-regression head per eligible task from the provided entries (in-memory; no file I/O). Task 2's out-of-fold scoring calls this directly with fold-split slices instead of going through Train. Entries that are cache hits or lack a correctness label are skipped. Tasks with fewer than minRows labelled rows are omitted from the returned Model.
func FitWithMinRows ¶
FitWithMinRows is Fit parameterized by the minimum labeled-row threshold. Production (Fit) uses minRows=60 (the emission gate); Task 2's out-of-fold validation passes a lower threshold so a head can still train on a smaller training fold (the paired-bootstrap CI is the real adoption guard). Behavior is otherwise identical to Fit.
func Load ¶
Load reads confhead-weights.json from path. Returns nil if the file is absent or cannot be parsed (caller treats nil as "no correctness signal").
Source Files