quality

package
v0.44.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package quality is the runnable spine of the "missing middle" validation ladder (epic #4509): the layer between primitive correctness tests (too local to catch a fluent-but-wrong decode) and end benchmarks (too coarse and late to localize an engine-caused regression).

The spine runs one versioned prompt case through a REFERENCE path and one fak ENGINE path, captures request/config/token/logit/output provenance, applies at least one deterministic comparator and one rubric scorer, and emits a machine-readable result with a pass/fail verdict and — on failure — a portable, replay-complete failure bundle. A quality claim is never green from a single stochastic sample: every oracle names the baseline it compares against and reports the FIRST divergence so a defect localizes to a token/step rather than being observed only in prose.

The package is deliberately stdlib-only and hermetic: the reference and engine paths are Runner adapters (case.go / runner.go), so the spine is CI-runnable without a live model, and real engine adapters wire in behind the same seam. The child cohort under #4509 extends this by ADDING its own oracle/runner files and registering them — it does not edit these cores.

Index

Constants

View Source 
const (
	FilingRegression  = "regression"
	FilingEvidenceGap = "evidence-gap"
)

Filing kinds. A regression is a localized failure with a replayable artifact; an evidence gap is an ABSENCE of trustworthy evidence. Both file, because the second is exactly the state a suite is most likely to mistake for a pass.

View Source 
const (
	PairwiseCandidate = "candidate"
	PairwiseBaseline  = "baseline"
	PairwiseTie       = "tie"
)

The pairwise winner vocabulary — a closed set shared by per-dimension and overall outcomes and by the `expect` the case declares.

View Source 
const (
	ResultSchema   = "fak-quality-result/1"
	ManifestSchema = "fak-quality-manifest/1"
)

ResultSchema and ManifestSchema are the versioned tags on the two machine-readable artifacts the spine emits. Consumers pin the major so a schema bump is a conscious migration, not a silent field drift (#4519).

View Source 
const CaseSchema = "fak-quality-case/1"

CaseSchema is the versioned envelope tag for a quality case. Bumping the trailing version is a breaking change to the case shape; readers reject an unknown major so a stale corpus can never masquerade as current.

View Source 
const DefaultGreenWindow = 2

DefaultGreenWindow is how many CONSECUTIVE green runs of a finding's own coordinates close it. Two, not one: a single green run after a failure is equally consistent with a fix and with a flake, and closing on a flake loses the issue that was the only record of the defect.

View Source 
const IssueFilingSchema = "fak-quality-filing/1"

IssueFilingSchema is the versioned tag on a filing plan, report, and tracker. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not a silent field drift.

View Source 
const MultiplicityDecisionSchema = "fak-quality-multiplicity/1"

MultiplicityDecisionSchema is the versioned tag on a multiplicity decision. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not a silent field drift.

View Source 
const NightlyMatrixSchema = "fak-quality-nightly-matrix/1"

NightlyMatrixSchema is the versioned tag on a matrix report. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule).

View Source 
const PostSelectionSchema = "fak-quality-postselection/1"

PostSelectionSchema is the versioned tag on a post-selection decision. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not a silent field drift.

View Source 
const RegressionBisectSchema = "fak-quality-bisect/1"

RegressionBisectSchema is the versioned tag on a bisect verdict. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not a silent field drift.

View Source 
const ReleaseGateSchema = "fak-quality-release/1"

ReleaseGateSchema is the versioned tag on a release-qualification decision. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not a silent field drift.

View Source 
const ReplaySchema = "fak-quality-replay/1"

ReplaySchema is the versioned envelope tag for a replay verdict. Consumers pin the major exactly as they do for the case, result, and manifest schemas.

View Source 
const SetComparabilitySchema = "fak-quality-comparability/1"

SetComparabilitySchema is the versioned tag on a comparability decision. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not a silent field drift.

View Source 
const StageAbstain = "unclassified"

StageAbstain is not a stage: it is the refusal to name one when the bundle's evidence supports no signature. It is deliberately spelled in the same field so a consumer cannot read a missing stage as an absent problem.

View Source 
const StopTruncationEOS = "<eos>"

StopTruncationEOS is the sentinel token the stop-truncation oracle treats as a hard stop / end-of-sequence marker: a faithful engine emits NOTHING after its first occurrence.

View Source 
const SuitePlanSchema = "fak-quality-suite-plan/1"

SuitePlanSchema is the versioned tag on a split plan. Consumers pin the major so a schema bump is a conscious migration (the #4519 house rule), not field drift.

View Source 
const ThresholdAuditSchema = "fak-quality-threshold-audit/1"

Variables

View Source 
var SchedPolicies = []string{"fcfs", "priority", "longest-first"}

SchedPolicies is the closed set of scheduler policies this child qualifies: fcfs (submission order), priority (highest Priority first), and longest-first (most Steps first). Sorts are stable so every policy's order is deterministic.

View Source 
var Stages = []string{
	"rubric",
	"transport",
	"tokenization",
	"normalization",
	"logits",
	"stops",
	"sampling",
	"cache",
}

Stages is the closed localization vocabulary, in the order the classifier probes it. Classify emits a member of this set or StageAbstain, never anything else, so a consumer may route on the value without a default arm.

Functions

func ArithmeticGroundTruth 

func ArithmeticGroundTruth(prior, current float64, denominator int) string

ArithmeticGroundTruth renders the period's figures as the canonical "key: value" block a report-arithmetic case carries in its Reference.Text: the prior and current values of the reported metric and the reporting denominator an "N of M" claim must agree with. Keeping ground truth inside the case preserves the spine's replay contract — the numbers a report is judged against travel with the case, not with ambient state.

func DefaultBudgets added in v0.42.0 

func DefaultBudgets() map[Tier]TierBudget

DefaultBudgets is the built-in cost split. PR is fast and CPU-only so it can gate every push; nightly extends the wall for statistics and corpora but stays on CPU; release is unbounded and admits accelerators for GPU parity and hardware review.

func ExpandMatrix added in v0.44.0 

func ExpandMatrix(spec MatrixSpec) ([]MatrixCase, []SuiteReject)

ExpandMatrix expands a spec into one canonical quality case per (model, backend, mode, slice) point. Every case is stamped to the NIGHTLY tier and inherits the spec's provenance, so each cell records model, tokenizer, engine/backend, its deterministic oracle or seed, the code revision, and the tolerance/baseline it is judged under.

Order is fixed — models, then backends, then modes, then slices — so a matrix replays cell-for-cell. Two points that collide on a case id (a repeated axis value in the spec) are refused as ambiguous rather than racing: one id cannot carry two coordinates.

func Explain 

func Explain(r Result) string

Explain renders a Result as human-readable first-failure localization (#4520): on pass it states what was verified; on failure it names the first failing oracle, the exact token index and the reference-vs-engine tokens there, and the STAGE of the serving path the bundle's evidence attributes that divergence to (or an explicit abstention when the evidence attributes it nowhere — see Classify). It is the `fak quality explain` body — the bridge from a machine verdict to "here is where, and in which layer, it first went wrong".

func ExplainBisect 

func ExplainBisect(r BisectResult) string

ExplainBisect renders a BisectResult as an operator readout, mirroring Explain (run.go) and ExplainRelease (release_gate.go): REGRESSION names the first bad revision, its good predecessor, the localized first divergence, and the probe/cost budget spent; CLEAN and INDETERMINATE state their reason plainly — an indeterminate verdict is surfaced as a refusal, never dressed up as a clean pass.

func ExplainFilingPlan added in v0.44.0 

func ExplainFilingPlan(p FilingPlan) string

ExplainFilingPlan renders a plan as the operator readout, mirroring Explain / ExplainMatrix / ExplainBisect: one line per lifecycle action with the key it acts on and the reason it was chosen, so "the nightly filed nothing new and closed one" is legible without opening the tracker.

func ExplainMatrix added in v0.44.0 

func ExplainMatrix(r MatrixReport) string

ExplainMatrix renders a report as the operator readout: the nightly tier's cost envelope, the cell tally, then each deduplicated issue with the coordinates it is attributed to, its replay handle, and every cell it covers. It mirrors Explain / ExplainPlan — the bridge from a machine report to "one bug, here is where it lives, here is the artifact that reproduces it".

func ExplainMultiplicity added in v0.44.0 

func ExplainMultiplicity(d MultiplicityDecision) string

ExplainMultiplicity renders a decision as an operator readout: the bound the family bought, every family-level refusal, each blocking cell strongest-first with its divergence and replay artifact, and the per-tier cost. It mirrors ExplainRelease (release_gate.go) and ExplainPlan (suite_split.go) — the bridge from a machine decision to "here is which cell to open first, and what the grid cost to learn it".

func ExplainPlan added in v0.42.0 

func ExplainPlan(p SuitePlan) string

ExplainPlan renders a SuitePlan as an operator readout: each suite with its case count and cost envelope, its ordered cases, then every rejected case with the budget or header it broke. It mirrors Explain / ExplainRelease — the bridge from a machine plan to "here is what each suite costs and why a case was left out".

func ExplainPostSelection added in v0.44.0 

func ExplainPostSelection(d PostSelectionDecision) string

ExplainPostSelection renders a PostSelectionDecision as an operator readout. It mirrors ExplainRelease (release_gate.go): the bridge from a machine verdict to "here is exactly what the search cost you". BOTH readings are always printed, on adjacent lines — collapsing them to one number is the confusion the layer exists to prevent, so the renderer is not allowed to drop either.

func ExplainRelease 

func ExplainRelease(d ReleaseDecision) string

ExplainRelease renders a ReleaseDecision as an operator readout: RELEASED with the qualified cases, or BLOCKED naming the first actionable divergence first and then every other blocking gate. It mirrors Explain (run.go) — the bridge from a machine verdict to "here is exactly what to fix before you can ship".

func ExplainReplay added in v0.44.0 

func ExplainReplay(v ReplayVerdict) string

ExplainReplay renders a replay verdict as the human half of the replay verb: the state, the failure the bundle claimed, the failure the replay observed, and — when the replay ran — the full first-failure localization of the replayed run, so an operator reads the same explanation they would have read at the original failure site.

func ExplainSetComparability added in v0.44.0 

func ExplainSetComparability(d ComparabilityDecision) string

ExplainSetComparability renders a ComparabilityDecision as an operator readout. It mirrors ExplainPostSelection (post_selection.go): the bridge from a machine verdict to "here is exactly which arm broke the comparison". The SET SIZE is always printed next to the verdict, because the failure mode this layer exists to catch is an operator reading a two-arm conclusion off a campaign that selected more.

func KVEvictionEngine 

func KVEvictionEngine(defect string) kvEvictionRunner

KVEvictionEngine returns an evicting engine runner with an optional injected recompute defect: "" recomputes evicted positions faithfully (parity holds); "lost-position" loses an evicted position's content; "stale-recompute" reconstructs evicted positions from the wrong offset. Both defects first corrupt the fold at the first step that attends to an evicted position, so the oracle fails at exactly kvEvictionFirstMissStep — the token that depended on it.

func Register 

func Register(o Oracle)

Register adds an oracle to the shared registry. It panics on a duplicate name so two children silently shadowing each other's oracle is a build-time failure, not a runtime surprise.

Types

type ActionCompleteness 

type ActionCompleteness struct{}

ActionCompleteness is the actionability oracle for executive reports (#4554): every risk or blocker a report surfaces must carry an OWNER and a NEXT ACTION, and — where the item itself declares a decision is needed — a DECISION ASK. A report that names a blocker with no owner is not status, it is ambient anxiety: nothing in it tells the reader who moves next or what they move on. Where material-omission (#4552) catches an item that was DROPPED, this oracle catches an item that was RAISED but left un-actionable.

Report items travel structured, as a JSON array of item objects in the engine trace's Text (the richer-than-Tokens seam the spine documents): 

[
  {"kind": "risk", "title": "vendor API deprecation in Q3",
   "owner": "dana", "next_action": "draft the migration plan by Friday"},
  {"kind": "blocker", "title": "staging migration blocked on DBA review",
   "owner": "lee", "next_action": "escalate the review queue",
   "needs_decision": true, "decision_ask": "approve contractor DBA hours"}
]

Rules (deterministic, documented):

  • Items of kind "risk" or "blocker" (case-insensitive) are ACTIONABLE and must carry a non-placeholder owner and next_action. Other kinds (wins, info, decisions already made) carry no such obligation here.
  • An actionable item with needs_decision=true must also carry a decision_ask — that is the "where applicable" clause: the item itself declares that a decision is being requested, so the ask must be stated.
  • A field that is empty, whitespace, or a placeholder ("TBD", "todo", "unassigned", "n/a", "none", "?") is MISSING — an owner of "TBD" is the canonical incomplete blocker, not an owner.

Score = complete actionable items / actionable items; Pass iff Score >= Rubric.MinScore (default 1: every raised risk/blocker must be actionable). On failure the Detail lists EVERY incomplete item with the exact field(s) it is missing, so the fix is a named list, not a hunch. Edge behavior (defined and tested): a report with no risk/blocker items passes vacuously; a Text that does not parse as a JSON item array fails closed with score 0 — an unparseable report cannot prove its blockers are owned.

func (ActionCompleteness) Judge 

func (ActionCompleteness) Judge(_ Trace, eng Trace, c QualityCase) Verdict

Judge parses the engine report's structured items and checks every risk and blocker for an owner, a next action, and — when the item flags needs_decision — a decision ask. Score is the fraction of actionable items that are complete; on failure Detail lists each incomplete item with the exact fields it is missing.

func (ActionCompleteness) Kind 

func (ActionCompleteness) Kind() string

func (ActionCompleteness) Name 

func (ActionCompleteness) Name() string

type ArmDecision added in v0.44.0 

type ArmDecision struct {
	Index    int     `json:"index"`
	ID       string  `json:"id"`
	Point    string  `json:"point"`
	P        float64 `json:"p_value"`
	Winner   bool    `json:"winner"`
	Admitted bool    `json:"admitted"`
	Reason   string  `json:"reason,omitempty"`
}

ArmDecision is one arm's adjudicated record: what it was, what it reported, whether it was the retained arm, and — when it could not be interpreted — why.

type ArmRole added in v0.44.0 

type ArmRole string

ArmRole is the closed set of roles an arm may play in a campaign. Typing the role is what makes the SHAPE of the comparison checkable: a campaign reads its treatments against a control, so exactly one control is required and at least one treatment must exist for there to be a comparison at all. 

const (
	// RoleControl is the held-out arm every treatment is read against. Exactly one
	// per campaign — two controls means the campaign never said which baseline its
	// deltas are measured from.
	RoleControl ArmRole = "control"
	// RoleTreatment is an arm that varies whatever the campaign set out to vary. A
	// campaign needs at least one, and every one of them is inside the comparability
	// check — including the ones an operator is not currently looking at.
	RoleTreatment ArmRole = "treatment"
)

type ArmRoleRecord added in v0.44.0 

type ArmRoleRecord struct {
	Index  int     `json:"index"`
	ID     string  `json:"id"`
	Role   ArmRole `json:"role"`
	Typed  bool    `json:"typed"`
	Reason string  `json:"reason,omitempty"`
}

ArmRoleRecord is one arm's adjudicated role: what it declared, and — when the role could not be typed — why. It is emitted for every handed arm so a reader can see the campaign's shape as the gate understood it.

type BaselineSpec 

type BaselineSpec struct {
	ID       string `json:"id"`
	Revision string `json:"revision"`
}

BaselineSpec pins the independently produced baseline artifact.

type BatchInvariance 

type BatchInvariance struct{}

BatchInvariance is the differential oracle for batch invariance (#4532): every placement in the engine's sweep must reproduce the reference (alone) token stream exactly. Any mismatch is a cross-request state leak, reported as the FIRST divergence with the offending BATCH POSITION named in Detail — so "the batched output looked off" localizes to "at batch position 1 of 3, token 2 was 'fjord' where the alone decode emitted 'dune'". The oracle fails closed: a trace carrying no sweep, or a sweep that never actually batches the target, cannot prove invariance and does not pass.

func (BatchInvariance) Judge 

func (BatchInvariance) Judge(ref, eng Trace, c QualityCase) Verdict

func (BatchInvariance) Kind 

func (BatchInvariance) Kind() string

func (BatchInvariance) Name 

func (BatchInvariance) Name() string

type BatchInvarianceRunner 

type BatchInvarianceRunner struct {
	Label string
	// contains filtered or unexported fields
}

BatchInvarianceRunner decodes the case's target request once per batch in the case's plan and reports the whole sweep. The zero value is a faithful engine (per-request decode is a pure function of the request, so every placement reproduces the alone decode); the defect field (set via BatchInvarianceEngine) injects a cross-request leak. Trace.Tokens carries the first placement's stream; Trace.Text carries the full sweep as JSON for the oracle.

func BatchInvarianceEngine 

func BatchInvarianceEngine(defect string) BatchInvarianceRunner

BatchInvarianceEngine returns a batched engine runner with an optional injected defect: "" decodes every placement faithfully (batch composition cannot reach the target's stream); "position-leak" makes the target's token at step 2 depend on its batch position; "neighbor-leak" makes it depend on a specific neighbor being scheduled before the target. These are the deterministic mutant sources the tests use to prove the invariance gate trips.

func (BatchInvarianceRunner) Name 

func (r BatchInvarianceRunner) Name() string

func (BatchInvarianceRunner) Run 

func (r BatchInvarianceRunner) Run(c QualityCase) (Trace, error)

type BisectOutcome 

type BisectOutcome string

BisectOutcome is the closed outcome of a lineage bisect. Found localizes a good→bad transition to a first-bad revision; Clean means every probed boundary is good (no regression in range); Indeterminate means the bisect could not produce a trustworthy answer — an unclassifiable point, an already-bad oldest point, or an empty lineage. Indeterminate is a first-class refusal, never a silent Clean. 

const (
	OutcomeFound         BisectOutcome = "found"
	OutcomeClean         BisectOutcome = "clean"
	OutcomeIndeterminate BisectOutcome = "indeterminate"
)

type BisectPoint 

type BisectPoint struct {
	Revision    string  `json:"revision"`          // lineage: commit / code-module revision
	Engine      string  `json:"engine"`            // lineage: engine/backend + config dimension
	Machine     string  `json:"machine,omitempty"` // lineage: machine/host class (held fixed per commit sweep)
	Tier        Tier    `json:"tier"`              // pr / nightly / release cadence this point runs at
	CostSeconds float64 `json:"cost_seconds"`      // runtime/resource cost to evaluate this point
}

BisectPoint is one coordinate in an ordered regression lineage: a commit/module revision, crossed with the engine/config dimension and machine class it is evaluated under, plus the tier and per-point cost of producing its evidence. The sequence is the "commit × config × machine" lineage the issue bisects; a single sweep bisects along ONE axis (typically the commit list) with the others held fixed, but the type carries all three so a config- or machine-axis sweep uses the same contract. Points are ordered oldest→newest, so the first good→bad transition is the first bad revision. It carries no Evidence itself: evidence is produced lazily by a Probe, so a long lineage costs only the points the bisect visits.

func DemoBisectLineage 

func DemoBisectLineage() []BisectPoint

DemoBisectLineage builds an ordered commit×config lineage for the spine demo case: seven synthetic revisions r0..r6 evaluated under one fixed engine/config on one machine class, oldest first. It is the hermetic fixture the witness test sweeps over — a stand-in for a real git commit range whose per-point evidence a driver would produce by checking out each revision and running the case.

type BisectResult 

type BisectResult struct {
	Schema          string         `json:"schema"`
	Outcome         BisectOutcome  `json:"outcome"`
	FirstBad        *BisectPoint   `json:"first_bad,omitempty"`
	LastGood        *BisectPoint   `json:"last_good,omitempty"`
	FirstDivergence *Divergence    `json:"first_divergence,omitempty"`
	Replay          *FailureBundle `json:"replay,omitempty"`
	Probes          int            `json:"probes"`
	CostSeconds     float64        `json:"cost_seconds"`
	Reason          string         `json:"reason"`
}

BisectResult is the machine-readable verdict of a lineage bisect: the outcome, the first-bad point and its good predecessor (on Found), the first actionable divergence and scrubbed replay bundle folded from the culprit's evidence, and the number of points actually probed plus their summed cost — the bisect-efficiency and runtime/resource-cost documentation (#4583 acceptance). It is a pure function of (lineage, Probe), so a verdict replays.

func Bisect 

func Bisect(lineage []BisectPoint, probe Probe) BisectResult

Bisect finds the first revision at which the case regresses across an ordered lineage, calling probe lazily and only at the points a binary search visits. It assumes the lineage is monotone (a good prefix then a bad suffix) — git-bisect's contract — and PROVES the two claims it makes: the reported first-bad point was probed Bad and its immediate predecessor was probed Good, a genuine, localized good→bad transition. Any point that probes indeterminate halts the search with an indeterminate outcome rather than being guessed past; an oldest point that is already bad, or a newest point that is still good, are reported honestly (regression precedes the range / no regression in range) instead of being forced into a culprit.

type CampaignArm added in v0.44.0 

type CampaignArm struct {
	ID   string            `json:"id"`
	Role ArmRole           `json:"role"`
	Axes map[string]string `json:"axes"`
}

CampaignArm is one SELECTED arm of a campaign: its id, the role it plays, and the axis values it witnessed. Axes maps a held-axis name to the value that arm ran under; an axis absent from the map (or present but blank) is UNWITNESSED, not matched. Every arm the campaign selected belongs here, including the ones outside the pair an operator happens to be reading — those are exactly the arms a pairwise check drops.

type CampaignDeclaration added in v0.44.0 

type CampaignDeclaration struct {
	ID       string   `json:"id"`
	HeldAxes []string `json:"held_axes"`
}

CampaignDeclaration is what a multi-arm campaign is held to: which axes it claims to hold constant across every selected arm. HeldAxes is the operator's assertion, and checking it against what the arms actually witnessed is the whole gate — a campaign that declares no held axis is refused rather than passed vacuously, because "nothing was held" is not the same claim as "everything matched".

type CaseMetadata 

type CaseMetadata struct {
	Model     Revision       `json:"model"`
	Tokenizer Revision       `json:"tokenizer"`
	Engine    EngineSpec     `json:"engine"`
	Code      Revision       `json:"code"`
	Oracle    OracleEvidence `json:"oracle"`
	Tolerance ToleranceSpec  `json:"tolerance"`
	Baseline  BaselineSpec   `json:"baseline"`
	Tier      TierSpec       `json:"tier"`
	Cost      CostSpec       `json:"cost"`
	// Owner is the accountable team or person for this case — a case with no
	// owner has no one to fix it when it fails, so #4574 refuses it. Family is
	// the evidence class the suite splitter routes on (see EvidenceFamily).
	Owner  string `json:"owner"`
	Family string `json:"family"`
}

CaseMetadata pins every external input needed to reproduce and route a case.

type CategoricalSampler 

type CategoricalSampler struct {
	// contains filtered or unexported fields
}

CategoricalSampler draws tokens from a fixed categorical distribution with a deterministic PRNG, via inverse-CDF lookup over the cumulative probabilities.

func NewCategoricalSampler 

func NewCategoricalSampler(tokens []string, probs []float64) (CategoricalSampler, error)

NewCategoricalSampler validates and normalizes the distribution. Negative probabilities, a zero-mass distribution, or a token/probability length mismatch are refused rather than sampled from.

func (CategoricalSampler) Sample 

func (s CategoricalSampler) Sample(n int, seed int64) []string

Sample draws n tokens with a PRNG seeded from seed. Same (n, seed), same sequence — the hermetic replay contract.

type CitationValidity 

type CitationValidity struct{}

CitationValidity is the citation MACHINERY oracle for executive reports (#4558): every citation marker the engine text carries must resolve to an evidence entry the case declares, and no marker may reference a non-existent/fabricated source. It is distinct from claim-grounding (#4551): that oracle judges whether the report's PROSE is backed by evidence content; this one judges whether the report's citation LINKS are wired — a report can be perfectly grounded and still cite [9] when only sources 1–3 exist, and a dangling citation is exactly how a fabricated source enters a rollup.

Citation markers are parsed from eng.Text by citeMarkerRE: `[` + an id + `]` where the id is either bare digits ("[1]") or letters, an optional single hyphen, then digits ("[S3]", "[E-12]"). Anything else in brackets ("[sic]", "[]", "[1.2]") is not a citation marker and is skipped, never panicked on. Ids compare case-insensitively ("[s3]" resolves to evidence "S3").

The allowed evidence ids travel in the case as Rubric.Required: each entry is one evidence id, optionally followed by ':' or whitespace and a link/description that is carried for humans but ignored for resolution — "1: https://ops/rollup-w28" declares id "1". CitationRubric is the helper constructor that builds such a spec.

Score = resolved markers / total markers (each occurrence counts). Pass iff Score >= Rubric.MinScore (default 1: no dangling marker). On failure Detail names the FIRST dangling citation — the fabricated source, localized. Declared evidence that is never cited is a soft warning appended to Detail, never a failure: the hard failure of this oracle is only the dangling marker.

Edge behavior (defined and tested): a report with zero citation markers has nothing to resolve and passes at score 1 with a Detail note (a case that REQUIRES citations should declare that via its grounding/omission rubric, which own must-appear content); markers judged against an empty evidence set all dangle and fail closed at score 0.

func (CitationValidity) Judge 

func (CitationValidity) Judge(_ Trace, eng Trace, c QualityCase) Verdict

func (CitationValidity) Kind 

func (CitationValidity) Kind() string

func (CitationValidity) Name 

func (CitationValidity) Name() string

type ClaimGrounding 

type ClaimGrounding struct{}

ClaimGrounding is the claim-level grounding oracle for executive reports (#4551): every claim sentence the engine text asserts must be supported by the case's evidence snippets. It inverts GroundingRubric's direction — that oracle checks the report CONTAINS the required phrases; this one checks the report asserts NOTHING the evidence does not back, which is the direction fabricated claims enter from.

Evidence travels in the case as Rubric.Required: this oracle reads each required entry as one allowed source snippet (not as a must-appear phrase). Claims are the ". "/newline-split sentences of eng.Text.

Grounding rule (deterministic, documented): a claim is GROUNDED iff a single evidence snippet contains at least groundingOverlap (60%) of the claim's distinct significant tokens. A significant token is a lowercased run of letters/digits/'%' that is at least 4 runes long or carries a digit — so figures like "12%" and "q3" always count while connective filler ("the", "and", "was") never decides grounding. A claim with no significant tokens asserts nothing checkable and is trivially grounded.

Score = grounded claims / total claims; Pass iff Score >= Rubric.MinScore (default 1: every claim must be grounded). On failure Detail names the FIRST ungrounded claim — localizing the hallucination, per the spine contract.

Edge behavior (defined and tested): an empty report asserts no claims and passes with score 1; a non-empty report judged against an empty evidence set fails closed with score 0 (an unsupported assertion is a hallucination, not a skipped check).

func (ClaimGrounding) Judge 

func (ClaimGrounding) Judge(_ Trace, eng Trace, c QualityCase) Verdict

func (ClaimGrounding) Kind 

func (ClaimGrounding) Kind() string

func (ClaimGrounding) Name 

func (ClaimGrounding) Name() string

type Classification added in v0.44.0 

type Classification struct {
	Stage  string `json:"stage"`
	Reason string `json:"reason"`
}

Classification is one first-failure localization: the stage the evidence points at and the reason naming the evidence it was read from. It is a pure function of the bundle, so it replays identically and is safe to embed in the emitted artifact.

func Classify added in v0.44.0 

func Classify(r Result) Classification

Classify localizes the first failure of a result. A passing result and a failing result with no bundle both abstain — the first because there is nothing to localize, the second because there is no evidence to localize from.

func (Classification) Abstained added in v0.44.0 

func (c Classification) Abstained() bool

Abstained reports whether the classifier declined to name a stage.

type ComparabilityDecision added in v0.44.0 

type ComparabilityDecision struct {
	Schema   string               `json:"schema"`
	Campaign string               `json:"campaign"`
	Verdict  ComparabilityVerdict `json:"verdict"`
	// Comparable is the single-boolean read of Verdict, false for both refusals.
	Comparable bool     `json:"comparable"`
	HeldAxes   []string `json:"held_axes"`
	// SetSize is the number of arms actually evaluated — the COMPLETE selected set,
	// which is what distinguishes this gate from a favored-pair check.
	SetSize  int                    `json:"set_size"`
	Controls int                    `json:"controls"`
	Arms     []ArmRoleRecord        `json:"arms"`
	Findings []ComparabilityFinding `json:"findings,omitempty"`
	Bound    string                 `json:"bound"`
}

ComparabilityDecision is the machine-readable output of the gate: the declaration, the arm set with every role typed, the verdict, and every finding that produced it. Comparable is true only for the SetComparable verdict — a consumer that reads nothing but that boolean still fails closed.

func EvaluateSetComparability added in v0.44.0 

func EvaluateSetComparability(decl CampaignDeclaration, arms []CampaignArm) ComparabilityDecision

EvaluateSetComparability adjudicates whether a campaign's complete selected arm set is comparable under its declaration. It types every arm's role, requires every selected arm to witness every declared held axis, and evaluates each axis over the WHOLE set rather than over any one pair.

It returns no error: every failure rides on the decision as a typed finding. It is a pure function of (declaration, arms) — arms are visited in the order handed over and axes in the order declared, so no map iteration reaches the output and the same inputs always produce the same decision, byte for byte.

type ComparabilityFinding added in v0.44.0 

type ComparabilityFinding struct {
	Code   ComparabilityFindingCode `json:"code"`
	Axis   string                   `json:"axis,omitempty"`
	Arms   []string                 `json:"arms,omitempty"`
	Detail string                   `json:"detail"`
}

ComparabilityFinding is one typed reason the gate did not certify comparability. Axis localizes it to the held axis at fault and Arms names the EXACT arms implicated, so a finding is actionable rather than a bare "not comparable".

type ComparabilityFindingCode added in v0.44.0 

type ComparabilityFindingCode string

ComparabilityFindingCode is the closed vocabulary of typed findings. A structured code is what lets a consumer branch on WHY comparability was withheld instead of pattern-matching prose. 

const (
	// FindingDeclarationInvalid: the campaign declares nothing the gate can hold it
	// to — no id, no held axes, or fewer than two selected arms.
	FindingDeclarationInvalid ComparabilityFindingCode = "campaign_declaration_invalid"
	// FindingArmRoleUntyped: an arm names no id, repeats an id already used, or
	// declares a role outside the closed set. The shape of the comparison is then
	// unknown.
	FindingArmRoleUntyped ComparabilityFindingCode = "arm_role_untyped"
	// FindingArmRolesUnbalanced: the typed roles do not form a comparison — not
	// exactly one control, or no treatment at all.
	FindingArmRolesUnbalanced ComparabilityFindingCode = "arm_roles_unbalanced"
	// FindingHeldAxisUnwitnessed: a selected arm hands over no value for a declared
	// held axis. Silence is not agreement.
	FindingHeldAxisUnwitnessed ComparabilityFindingCode = "held_axis_unwitnessed"
	// FindingHeldAxisDiffers: a declared held axis takes more than one value across
	// the selected set. This is the set-wide defect a favored-pair check misses.
	FindingHeldAxisDiffers ComparabilityFindingCode = "held_axis_differs"
)

type ComparabilityVerdict added in v0.44.0 

type ComparabilityVerdict string

ComparabilityVerdict is the closed set of conclusions the gate may reach. The three-way split is load-bearing: collapsing could-not-establish into not-comparable would report an absence of evidence as a proven difference, and collapsing it into comparable would report it as a pass. 

const (
	// SetComparable: every held axis was witnessed by every selected arm and every
	// arm agreed on it. This is the only verdict that licenses reading the campaign
	// as a controlled comparison.
	SetComparable ComparabilityVerdict = "comparable"
	// SetNotComparable: every required witness was present and at least one held axis
	// takes more than one value across the set. The campaign is PROVABLY confounded,
	// and the finding names which axis and which arms.
	SetNotComparable ComparabilityVerdict = "not_comparable"
	// SetCouldNotEstablish: some required evidence is unknown — an unwitnessed axis,
	// an untyped arm role, or a declaration that names nothing to hold. Never a pass,
	// and never upgraded to a proven difference.
	SetCouldNotEstablish ComparabilityVerdict = "could_not_establish"
)

type ComparisonDecision added in v0.44.0 

type ComparisonDecision struct {
	Index           int            `json:"index"`
	Cell            string         `json:"cell"`
	Model           string         `json:"model"`
	Slice           string         `json:"slice"`
	Metric          string         `json:"metric"`
	CaseID          string         `json:"case_id"`
	Primary         bool           `json:"primary"`
	Tier            Tier           `json:"tier"`
	CostSeconds     float64        `json:"cost_seconds"`
	P               float64        `json:"p_value"`
	Adjusted        float64        `json:"adjusted_p"`
	Rejected        bool           `json:"rejected"`
	Gated           bool           `json:"gated"`
	State           EvidenceState  `json:"state"`
	Reason          string         `json:"reason"`
	FirstDivergence *Divergence    `json:"first_divergence,omitempty"`
	Replay          *FailureBundle `json:"replay,omitempty"`
}

ComparisonDecision is one comparison's adjudicated outcome: its adjusted p-value, whether the correction rejected it, whether the hierarchical gate left it untested, and — when it blocks — the reason plus the divergence and replay artifact an operator acts on. Rejected is the PURE statistical outcome and is never overwritten by an administrative refusal, so a null simulation can read the procedure's own error rate off it.

type ConfidenceCalibration 

type ConfidenceCalibration struct{}

ConfidenceCalibration is the confidence/uncertainty calibration oracle for executive reports (#4555): every claim a report asserts states a confidence, and the case's reference carries the GROUND-TRUTH support flag for each claim. Calibration means the stated confidence never exceeds what the evidence licenses — a high-confidence claim must not be one that is actually unsupported, and a weak-evidence claim must be hedged. This is orthogonal to grounding (claim-grounding checks a claim HAS evidence; this oracle checks the report's certainty LANGUAGE matches the evidence it has).

Payload contract (both hermetic, JSON-in-Trace per the additive-seam rule): 

eng.Text: [{"claim":"...","confidence":"high|medium|low"}, ...]
ref.Text: [{"claim":"...","support":"strong|weak|none"}, ...]

Calibration rule (deterministic, documented): each support level fixes a CEILING on the confidence a calibrated report may state for that claim: 

strong -> high   (any confidence is calibrated; hedging harder is safe)
weak   -> low    (a weak-evidence claim must be hedged)
none   -> low    (an unsupported claim may at most be floated as hedged
                  speculation; asserting it at medium/high confidence is
                  the overconfident-fabrication class this oracle exists
                  to catch)

A claim with no ground-truth record is treated as support "none" — fail closed: unverifiable support never licenses confidence. A claim stating an unknown confidence token is itself miscalibrated (uncheckable certainty is not calibrated certainty).

Score = calibrated claims / total claims; Pass iff Score >= Rubric.MinScore (default 1: every claim must be calibrated). On failure Detail names the FIRST miscalibrated claim and its failure class — localizing the overconfidence, per the spine contract.

Edge behavior (defined and tested): a report asserting no claims has nothing miscalibrated and passes at score 1; an unparseable claim or support payload fails closed at score 0 (a payload that cannot be checked is not a green payload).

func (ConfidenceCalibration) Judge 

func (ConfidenceCalibration) Judge(ref, eng Trace, c QualityCase) Verdict

func (ConfidenceCalibration) Kind 

func (ConfidenceCalibration) Kind() string

func (ConfidenceCalibration) Name 

func (ConfidenceCalibration) Name() string

type CostSpec 

type CostSpec struct {
	RuntimeSeconds int64 `json:"runtime_seconds"`
	TimeoutSeconds int64 `json:"timeout_seconds"`
	CPU            int   `json:"cpu"`
	MemoryMiB      int64 `json:"memory_mib"`
	Accelerators   int   `json:"accelerators,omitempty"`
}

CostSpec documents expected runtime and peak resource requirements. TimeoutSeconds is the hard wall a run is killed at — distinct from the expected RuntimeSeconds — so a hung case cannot silently hold a suite's budget open. #4574 requires every case to declare it.

type DistributionTV 

type DistributionTV struct{}

DistributionTV is the distribution-comparison oracle (#4530). The case's Reference trace carries the target distribution — the vocabulary in Reference.Tokens and the aligned target probabilities as the single row Reference.Logits[0] — and the engine trace carries the N drawn samples in its Tokens. Judge builds the empirical histogram over the reference vocabulary, computes the total-variation distance to the target (any out-of-vocabulary sample mass counts fully against the engine), and passes iff TV <= distributionTVThreshold. Score is 1 - TV.

func (DistributionTV) Judge 

func (DistributionTV) Judge(ref, eng Trace, _ QualityCase) Verdict

func (DistributionTV) Kind 

func (DistributionTV) Kind() string

func (DistributionTV) Name 

func (DistributionTV) Name() string

type Divergence 

type Divergence struct {
	Index     int    `json:"index"`
	Reference string `json:"reference"`
	Engine    string `json:"engine"`
}

Divergence is the first step at which two token streams disagree: its index and the reference vs engine token there. It is what turns "the report looked wrong" into "token 7 was 'increased' where the reference emitted 'decreased'".

type DtypeDelta 

type DtypeDelta struct{}

DtypeDelta is the dtype-parity differential oracle (#4541): every declared dtype lane in the engine payload must stay within its OWN band of the FP32 reference logits. Lanes are judged in the fixed dtSpecs order, tokens ascending, and the first out-of-band logit is reported as the first divergence with the offending dtype and token named — "FP16 looked off" becomes "dtype fp16 diverged beyond its band at token 4".

func (DtypeDelta) Judge 

func (DtypeDelta) Judge(ref, eng Trace, _ QualityCase) Verdict

func (DtypeDelta) Kind 

func (DtypeDelta) Kind() string

func (DtypeDelta) Name 

func (DtypeDelta) Name() string

type EffectiveRequest added in v0.44.0 

type EffectiveRequest struct {
	Prompt string         `json:"prompt"`
	Params SamplingParams `json:"params"`
	// Unsupported names request fields (from the requestFields vocabulary) this
	// runner cannot honor. Naming a field here is not by itself a failure: it
	// costs a run nothing unless the case actually specifies that field.
	Unsupported []string `json:"unsupported,omitempty"`
}

EffectiveRequest is a runner's declaration of the request it ACTUALLY executes for a case: the prompt and params it ran with, plus the field names it cannot honor at all. Unsupported is a static capability claim — a real adapter always has some field it does not implement — while Prompt/Params are what this specific case turned into on the way in.

type EngineSpec 

type EngineSpec struct {
	Name    string            `json:"name"`
	Backend string            `json:"backend"`
	Flags   map[string]string `json:"flags,omitempty"`
}

EngineSpec identifies the implementation and backend plus replay-affecting flags.

type Evidence 

type Evidence struct {
	CaseID          string             `json:"case_id"`
	State           EvidenceState      `json:"state"`
	Provenance      EvidenceProvenance `json:"provenance"`
	FirstDivergence *Divergence        `json:"first_divergence,omitempty"`
	Replay          *FailureBundle     `json:"replay,omitempty"`
	Detail          string             `json:"detail,omitempty"`
}

Evidence is a produced qualification record for one case: its state, provenance, and — on a non-pass — the localized first divergence and scrubbed replay bundle folded from the underlying quality Result. It is what a producer submits to the release gate; the gate never re-runs a case, it adjudicates the submitted evidence.

func EvidenceFromResult 

func EvidenceFromResult(prov EvidenceProvenance, r Result) Evidence

EvidenceFromResult folds a spine Result (run.go) into release Evidence under a given provenance. It is the seam that ties the release gate to the package's own gates: a RunCase pass becomes releasing evidence, and a RunCase failure becomes blocking Fail evidence carrying the SAME first-divergence and scrubbed FailureBundle the spine already localized. Provenance (model/revision/…) is supplied by the producer — the Result itself is deliberately host- and clock-free, so attribution is added here at submission time.

type EvidenceFamily added in v0.42.0 

type EvidenceFamily string

EvidenceFamily is the class of evidence a case produces. #4574 scope names the five families top stacks separate; the splitter records the family so an operator can see WHAT a suite is qualifying, not just how long it costs. 

const (
	// FamilyDeterministic is an exact-oracle / greedy differential check: cheap,
	// reproducible from a fixed oracle, the natural per-PR gate.
	FamilyDeterministic EvidenceFamily = "deterministic"
	// FamilyGPUParity compares device backends and needs an accelerator, so it can
	// never ride the CPU-only PR lane.
	FamilyGPUParity EvidenceFamily = "gpu_parity"
	// FamilyStatistics is a distribution/sampling check that needs many samples —
	// too slow for a PR, the natural nightly tenant.
	FamilyStatistics EvidenceFamily = "statistics"
	// FamilyCorpora replays a task corpus: broad, slow, nightly-or-release.
	FamilyCorpora EvidenceFamily = "corpora"
	// FamilyReview is a rubric/judge review of report quality: release-cadence
	// qualification, not a per-PR blocker.
	FamilyReview EvidenceFamily = "review"
)

type EvidenceProvenance 

type EvidenceProvenance struct {
	Model     string `json:"model"`
	Tokenizer string `json:"tokenizer"`
	Engine    string `json:"engine"`
	// Seed pins a stochastic case; Oracle names the deterministic comparator. At
	// least one must be set — a case that is neither seeded nor oracle-judged is not
	// reproducible, so it cannot qualify a release.
	Seed   int64  `json:"seed,omitempty"`
	Oracle string `json:"oracle,omitempty"`
	// Revision is the code/module revision the evidence was produced at. It is the
	// staleness key: evidence produced at a different revision than the release
	// under test is stale and blocks.
	Revision string `json:"revision"`
	// Baseline is the tolerance/baseline provenance the case was judged against.
	Baseline string `json:"baseline"`
}

EvidenceProvenance is the per-case provenance #4578 requires every qualification record to carry: which model/tokenizer/engine produced the evidence, the seed (stochastic) OR deterministic oracle it was judged by, the code/module revision it was produced at, and the tolerance/baseline it was compared against. Absent provenance is treated as inconclusive — evidence you cannot attribute cannot release.

type EvidenceState 

type EvidenceState string

EvidenceState is the closed set a required gate's evidence can be in. Only Pass releases; Fail, Missing, Stale, and Inconclusive all BLOCK — "no evidence" and "unclear evidence" are never a pass (#4578 acceptance: missing or inconclusive evidence is never pass). 

const (
	StatePass         EvidenceState = "pass"
	StateFail         EvidenceState = "fail"
	StateMissing      EvidenceState = "missing"
	StateStale        EvidenceState = "stale"
	StateInconclusive EvidenceState = "inconclusive"
)

type FailureBundle 

type FailureBundle struct {
	CaseID          string      `json:"case_id"`
	Case            QualityCase `json:"case"`
	FailingOracle   string      `json:"failing_oracle"`
	FailingKind     string      `json:"failing_kind"`
	FirstDivergence *Divergence `json:"first_divergence,omitempty"`
	Reference       Trace       `json:"reference"`
	Engine          Trace       `json:"engine"`
	Detail          string      `json:"detail"`
	Scrubbed        bool        `json:"scrubbed"`
	// Requests travels with the bundle so a failure reproduces from the bundle
	// ALONE (#4515): a replayer that cannot see the engine dropped top_k would
	// re-run a different request than the one that failed.
	Requests RequestRecord `json:"requests"`
	// Classification is the machine-readable half of first-failure localization
	// (#4520): which layer of the serving path the bundle's own evidence points
	// at, or an explicit abstention when it points nowhere. It is Classify's
	// output, stamped here so a CI consumer routes on the artifact instead of
	// re-deriving it from prose.
	Classification *Classification `json:"classification,omitempty"`
}

FailureBundle is the portable, replay-complete artifact emitted on any failing run (#4515). It embeds the full case so the failure reproduces from the bundle alone, names the first oracle that failed, and pins the first token divergence when one exists. It is scrubbed by construction — it carries only case text the author already committed, never ambient secrets.

func LoadBundle added in v0.44.0 

func LoadBundle(data []byte) (FailureBundle, error)

LoadBundle decodes a failure bundle from JSON. It accepts either a bare bundle or the full Result that `fak quality run --json` emits, so the artifact CI already stores replays without being unwrapped by hand. Unknown fields and trailing documents are refused: a bundle from a schema the reader does not understand must not be replayed as if it were understood.

func (FailureBundle) ReplayComplete added in v0.44.0 

func (f FailureBundle) ReplayComplete() error

ReplayComplete reports whether the bundle carries everything Replay needs, and names the first missing piece when it does not. It is the admission gate for a replay: a bundle that is short of evidence is refused with a reason rather than replayed to a misleading verdict.

type FieldDelta added in v0.44.0 

type FieldDelta struct {
	Field     string `json:"field"`
	Requested string `json:"requested"`
	Effective string `json:"effective"`
}

FieldDelta is one normalized request field whose executed value differed from the one the case declared — the request-level analogue of a token Divergence.

type Filer added in v0.44.0 

type Filer func(Filing) error

Filer writes ONE filing to the issue tracker of record — the single impure step of the auto-filer, and the seam a `gh issue create/comment/close` driver plugs into. A returned error means the filing did not land.

type Filing added in v0.44.0 

type Filing struct {
	Action FilingAction   `json:"action"`
	Key    RegressionKey  `json:"key"`
	Marker string         `json:"marker"`
	Kind   string         `json:"kind"`
	Title  string         `json:"title"`
	Body   string         `json:"body,omitempty"`
	Tier   Tier           `json:"tier,omitempty"`
	Cost   CostSpec       `json:"cost"`
	Replay *FailureBundle `json:"replay,omitempty"`
	Reason string         `json:"reason"`
	// contains filtered or unexported fields
}

Filing is one lifecycle action against ONE keyed issue: what to do, which issue (by marker), the rendered title and body, the tier and cost the finding is assigned to, and — on a regression — the scrubbed replay artifact the body embeds. Body is empty on a Hold, which files nothing.

type FilingAction added in v0.44.0 

type FilingAction string

FilingAction is the closed lifecycle vocabulary. Open and Update and Close are the three things a driver does to the tracker of record; Hold is the internal bookkeeping step of a green run that has not yet completed the window, surfaced so an operator can see recovery in progress rather than silence. 

const (
	ActionOpen   FilingAction = "open"
	ActionUpdate FilingAction = "update"
	ActionHold   FilingAction = "hold"
	ActionClose  FilingAction = "close"
)

type FilingFailure added in v0.44.0 

type FilingFailure struct {
	Filing Filing `json:"filing"`
	Error  string `json:"error"`
}

FilingFailure is one filing the driver could not land, with the error it returned. Its issue's lifecycle state is left untouched, so the next run proposes it again: a dropped file is retried, never mistaken for filed.

type FilingPlan added in v0.44.0 

type FilingPlan struct {
	Schema  string    `json:"schema"`
	Run     FilingRun `json:"run"`
	Filings []Filing  `json:"filings,omitempty"`
}

FilingPlan is the machine-readable output of folding one run: the run it was computed for and every lifecycle action it implies, in marker order so a plan is stable across runs. It is a pure function of (tracker state, run, observations) — same inputs, same plan — so a filing decision replays.

type FilingReport added in v0.44.0 

type FilingReport struct {
	Schema string          `json:"schema"`
	Run    FilingRun       `json:"run"`
	Landed []Filing        `json:"landed,omitempty"`
	Failed []FilingFailure `json:"failed,omitempty"`
}

FilingReport is what actually happened when a plan was applied.

type FilingRun added in v0.44.0 

type FilingRun struct {
	ID       string `json:"id"`
	Revision string `json:"revision"`
}

FilingRun identifies one run of a quality suite: the run's own id (what an operator cites) and the code/module revision it qualified. The revision is load-bearing, not decorative — evidence attributed to a different revision than the run is stale, and a run that names no revision can attribute nothing, so both are evidence gaps rather than passes.

type GateKind 

type GateKind string

GateKind is the evidence family a required case belongs to. #4578 requires the four families top stacks separate: deterministic (exact oracle), statistical (distribution/tolerance), hardware (device-dependent), and report (rubric). 

const (
	KindDeterministic GateKind = "deterministic"
	KindStatistical   GateKind = "statistical"
	KindHardware      GateKind = "hardware"
	KindReport        GateKind = "report"
)

type GreedyTokenDiff 

type GreedyTokenDiff struct{}

GreedyTokenDiff is the deterministic comparator: for a temperature-zero / greedy case, the engine token stream must equal the reference token stream exactly, and the first mismatch (or a length difference) is reported as the first divergence. This is the spine realization of #4522 (compare greedy decode token by token BEFORE text scoring) — a fluent report built from one wrong token is caught here, not left for a downstream text metric to average away.

func (GreedyTokenDiff) Judge 

func (GreedyTokenDiff) Judge(ref, eng Trace, _ QualityCase) Verdict

func (GreedyTokenDiff) Kind 

func (GreedyTokenDiff) Kind() string

func (GreedyTokenDiff) Name 

func (GreedyTokenDiff) Name() string

type GroundingRubric 

type GroundingRubric struct{}

GroundingRubric is the deterministic rubric scorer: it scores the engine text on the fraction of the case's required phrases present, fails if any forbidden claim appears, and gates on the case's MinScore (default 1: every required phrase must be present). It is the spine stand-in for the report-quality rubric layer (#4550–#4565): even without a calibrated judge, an executive report that silently omits a material required claim, or invents a forbidden one, fails a gate instead of being merely observed in prose.

func (GroundingRubric) Judge 

func (GroundingRubric) Judge(_ Trace, eng Trace, c QualityCase) Verdict

func (GroundingRubric) Kind 

func (GroundingRubric) Kind() string

func (GroundingRubric) Name 

func (GroundingRubric) Name() string

type IncrementalUnicode 

type IncrementalUnicode struct{}

IncrementalUnicode is the incremental Unicode / byte-fallback decoding oracle (#4533): token-by-token (streaming) detokenization must reconstruct exactly the same valid UTF-8 text as a full decode, even when a multibyte codepoint is split across two tokens and when byte-fallback tokens (raw bytes spelled "<0xE2>"-style) reassemble a codepoint. Tokens are modeled as byte fragments; the reference Text is the full/correct decoded string. The oracle asserts the engine's incrementally-assembled Text equals that reference decode and that no U+FFFD appears at a boundary a faithful decoder would have buffered. On mismatch, FirstDivergence.Index is the token index where decoding first diverged and Detail describes the bad boundary.

func (IncrementalUnicode) Judge 

func (IncrementalUnicode) Judge(ref, eng Trace, _ QualityCase) Verdict

func (IncrementalUnicode) Kind 

func (IncrementalUnicode) Kind() string

func (IncrementalUnicode) Name 

func (IncrementalUnicode) Name() string

type KVEvictionParity 

type KVEvictionParity struct{}

KVEvictionParity is the differential oracle for KV eviction+recomputation (#4535): the evicting engine's token stream must equal the never-evicted reference stream exactly, because a faithful recompute rebuilds each evicted position bit-for-bit. Any mismatch — a lost position, a miscomputed recompute, a truncated decode — is reported as the FIRST divergence, so "eviction corrupted the decode" localizes to the first token that depended on the bad position.

func (KVEvictionParity) Judge 

func (KVEvictionParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (KVEvictionParity) Kind 

func (KVEvictionParity) Kind() string

func (KVEvictionParity) Name 

func (KVEvictionParity) Name() string

type LogprobParity 

type LogprobParity struct{}

LogprobParity is the differential oracle for #4524: every returned logprob row — prompt echo and generation alike — must be ALIGNED (row i scores token i, one row per token, same candidate width as the reference) and within lpTolerance of the reference values. The first offending row is reported as the first divergence with a classified Detail, so "the logprobs look off" localizes to "token 1's row is token 0's row: off-by-one" or "row 0 carries a +2.48 constant offset: raw logits".

func (LogprobParity) Judge 

func (LogprobParity) Judge(ref, eng Trace, c QualityCase) Verdict

func (LogprobParity) Kind 

func (LogprobParity) Kind() string

func (LogprobParity) Name 

func (LogprobParity) Name() string

type LogprobParityRunner 

type LogprobParityRunner struct {
	Label string
	// contains filtered or unexported fields
}

LogprobParityRunner is the engine-path adapter for #4524: it computes the faithful prompt+generation trace and then applies an optional injected logprob defect. The token text is left byte-identical in every defect mode — exactly the property that makes these bugs invisible to token-differential oracles and the reason this oracle reads Logits.

func LogprobParityEngine 

func LogprobParityEngine(defect string) LogprobParityRunner

LogprobParityEngine returns an engine runner with an optional injected logprob defect: "" reports faithful aligned, normalized logprobs; "logprob-shift" shifts every row by one position (off-by-one alignment); "raw-logits" reports raw pre-softmax logits (wrong normalization). This is the deterministic mutant source the tests use to prove the gate trips.

func (LogprobParityRunner) Name 

func (r LogprobParityRunner) Name() string

func (LogprobParityRunner) Run 

func (r LogprobParityRunner) Run(c QualityCase) (Trace, error)

type MaterialItem 

type MaterialItem struct {
	Category string `json:"category,omitempty"`
	Text     string `json:"text"`
}

MaterialItem is one piece of material content a report must cover: its text (matched case-insensitively as a substring of the engine output) and an optional category used when reporting an omission.

type MaterialItems 

type MaterialItems struct {
	Wins      []string
	Risks     []string
	Blockers  []string
	Decisions []string
}

MaterialItems is the categorized declaration of everything material a report must cover: wins, risks, blockers, and decisions. It is the authoring surface for material-omission cases — build one, then carry it on the case via Rubric.

func (MaterialItems) Required 

func (m MaterialItems) Required() []string

Required flattens the categorized items into Rubric.Required entries using the "category: item" encoding materialItems parses back out. Order is deterministic: wins, risks, blockers, decisions, each in declaration order. Empty item strings are dropped — an empty item is not material content.

func (MaterialItems) Rubric 

func (m MaterialItems) Rubric(minScore float64) RubricSpec

Rubric is the helper constructor for a case judged by material-omission: it carries the categorized items in Rubric.Required and the pass threshold in MinScore (0 means the default — nothing material may be omitted). A case that ALSO names grounding-rubric should carry plain entries instead, since grounding matches each Required entry literally, category prefix and all.

type MaterialOmission 

type MaterialOmission struct{}

MaterialOmission is the omission-side complement to GroundingRubric (#4552): where grounding catches FABRICATED claims, this oracle catches DROPPED ones. A status report that reads fluently but silently omits a material win, risk, blocker, or decision must fail a gate, not merely read fine. The case declares its material items via Rubric.Required — either plain must-mention entries, or categorized entries built by MaterialItems.Rubric — and the oracle scores the fraction present in the engine text; Pass iff Score >= Rubric.MinScore (default 1: nothing material may be omitted). On failure the Detail names EVERY omitted item (with its category when declared), so the fix is a list, not a hunch.

func (MaterialOmission) Judge 

func (MaterialOmission) Judge(_ Trace, eng Trace, c QualityCase) Verdict

func (MaterialOmission) Kind 

func (MaterialOmission) Kind() string

func (MaterialOmission) Name 

func (MaterialOmission) Name() string

type MatrixCase added in v0.44.0 

type MatrixCase struct {
	Cell MatrixCell  `json:"cell"`
	Case QualityCase `json:"case"`
}

MatrixCase is one expanded cell: its coordinates and the canonical quality case those coordinates produce.

type MatrixCell added in v0.44.0 

type MatrixCell struct {
	CaseID  string `json:"case_id"`
	Model   string `json:"model"`
	Backend string `json:"backend"`
	Mode    string `json:"mode"`
	Slice   string `json:"slice"`
}

MatrixCell is one point in the matrix: the case id it expands to and its four axis coordinates. It is the unit an operator reads a failure in.

type MatrixCellResult added in v0.44.0 

type MatrixCellResult struct {
	Cell      MatrixCell       `json:"cell"`
	Outcome   MatrixOutcome    `json:"outcome"`
	Signature *MatrixSignature `json:"signature,omitempty"`
	Detail    string           `json:"detail"`
	// contains filtered or unexported fields
}

MatrixCellResult is one cell's outcome plus, when it did not pass, the signature it folds under and the detail that localized it.

type MatrixEngine added in v0.44.0 

type MatrixEngine func(cell MatrixCell, c QualityCase) (Runner, error)

MatrixEngine resolves the engine runner for one cell. A real harness returns an adapter bound to that model/backend/mode build; a returned error is not a failure of the code under test but an absence of evidence, so RunMatrix records it as inconclusive rather than as a pass or a fail.

type MatrixIssue added in v0.44.0 

type MatrixIssue struct {
	ID        string          `json:"id"`
	Kind      string          `json:"kind"` // "regression" or "inconclusive"
	Signature MatrixSignature `json:"signature"`
	// Attribution is the axis coordinates shared by every affected cell that no
	// passing cell holds — the narrowest coordinate the evidence indicts. It is
	// empty when the failures span the matrix with no common exonerated axis, and
	// saying so is the honest answer.
	Attribution []string     `json:"attribution,omitempty"`
	Cells       []MatrixCell `json:"cells"`
	Detail      string       `json:"detail"`
	// Replay is the representative scrubbed failure bundle: it embeds the case and
	// both traces, so this issue reproduces from the artifact alone. It is absent
	// only on an inconclusive issue, where by definition no artifact was produced.
	Replay *FailureBundle `json:"replay,omitempty"`
}

MatrixIssue is ONE deduplicated finding: every cell that observed a single defect signature, the axis coordinates that defect is attributed to, and one representative scrubbed replay artifact. Filing this — rather than one issue per failing cell — is the #4577 acceptance criterion.

type MatrixModel added in v0.44.0 

type MatrixModel struct {
	Model     Revision `json:"model"`
	Tokenizer Revision `json:"tokenizer"`
}

MatrixModel is one representative model on the model axis, with the tokenizer it is decoded under. The two travel together because a tokenizer swap changes the token stream every differential oracle compares — recording the model without its tokenizer would make a divergence unattributable.

type MatrixOutcome added in v0.44.0 

type MatrixOutcome string

MatrixOutcome is the closed set of states one cell can end in. There is deliberately no "skipped": a cell that did not produce evidence is Inconclusive, which blocks exactly like a failure. 

const (
	MatrixPass         MatrixOutcome = "pass"
	MatrixFail         MatrixOutcome = "fail"
	MatrixInconclusive MatrixOutcome = "inconclusive"
)

type MatrixReport added in v0.44.0 

type MatrixReport struct {
	Schema   string             `json:"schema"`
	Tier     Tier               `json:"tier"`
	Suite    Suite              `json:"suite"`
	Cells    []MatrixCellResult `json:"cells"`
	Issues   []MatrixIssue      `json:"issues,omitempty"`
	Rejected []SuiteReject      `json:"rejected,omitempty"`
}

MatrixReport is the machine-readable outcome of one nightly matrix run: the nightly suite the cells were routed into (carrying the tier's summed cost envelope), every cell's outcome, the deduplicated issues, and every cell the router refused.

func RunMatrix added in v0.44.0 

func RunMatrix(spec MatrixSpec, engine MatrixEngine, budgets map[Tier]TierBudget) MatrixReport

RunMatrix expands, routes, runs, and folds one nightly matrix. Budgets nil means DefaultBudgets. It never returns an error: every way a matrix can go wrong — an unroutable cell, an unavailable engine, a failing oracle — is a state in the report a gate can route on, and only MatrixReport.Green is a pass.

func (MatrixReport) Green added in v0.44.0 

func (r MatrixReport) Green() (bool, string)

Green is the fail-closed verdict on a matrix report, naming the first blocking reason. A refused cell blocks (refusing it is the point), any issue blocks (including an inconclusive one — missing evidence is never a pass), and a matrix that ran no cell at all blocks, because a matrix that qualified nothing has produced no evidence.

type MatrixSignature added in v0.44.0 

type MatrixSignature struct {
	FailingOracle string `json:"failing_oracle,omitempty"`
	FailingKind   string `json:"failing_kind"`
	Stage         string `json:"stage"`
}

MatrixSignature is the identity of a defect as the matrix sees it: which oracle failed first, of which kind, and which serving stage the bundle's own evidence attributes it to. Cells sharing a signature are the SAME defect observed at different coordinates, which is what makes the fold a deduplication rather than a summary. On an inconclusive cell, Kind is "inconclusive" and Stage is the cause the evidence went missing.

func (MatrixSignature) String added in v0.44.0 

func (s MatrixSignature) String() string

String renders the signature as the one line that names the defect.

type MatrixSlice added in v0.44.0 

type MatrixSlice struct {
	Name      string         `json:"name"`
	Prompt    string         `json:"prompt"`
	Params    SamplingParams `json:"params"`
	Reference Trace          `json:"reference"`
	Oracles   []string       `json:"oracles"`
	Rubric    RubricSpec     `json:"rubric,omitempty"`
}

MatrixSlice is one task slice: the prompt, the decode configuration, the reference trace that slice is judged against, and the oracles to apply. Slices are the "task" axis — a matrix that varies only the hardware proves nothing about whether a model still answers the questions it is deployed to answer.

type MatrixSpec added in v0.44.0 

type MatrixSpec struct {
	// Engine is the engine implementation under test; Backends and Modes are the
	// supported backends and engine modes it is exercised across.
	Engine   string        `json:"engine"`
	Models   []MatrixModel `json:"models"`
	Backends []string      `json:"backends"`
	Modes    []string      `json:"modes"`
	Slices   []MatrixSlice `json:"slices"`

	Code      Revision       `json:"code"`
	Oracle    OracleEvidence `json:"oracle"`
	Tolerance ToleranceSpec  `json:"tolerance"`
	Baseline  BaselineSpec   `json:"baseline"`
	Owner     string         `json:"owner"`
	Family    string         `json:"family"`
	// Cost is the per-cell runtime and resource declaration. It is checked against
	// the nightly tier budget by SplitSuites, so a cell too expensive for the
	// nightly wall is refused with the ceiling it broke instead of quietly making
	// the nightly overrun.
	Cost CostSpec `json:"cost"`
}

MatrixSpec declares one nightly matrix: the four axes plus the provenance every expanded cell inherits. The provenance fields are spec-level on purpose — a matrix qualifies ONE code revision against ONE baseline under ONE tolerance policy, so letting a cell carry its own would let a green matrix be assembled from cells that were never comparable.

type MetricComparison added in v0.44.0 

type MetricComparison struct {
	Model       string   `json:"model"`
	Slice       string   `json:"slice"`
	Metric      string   `json:"metric"`
	P           float64  `json:"p_value"`
	Tier        Tier     `json:"tier"`
	CostSeconds float64  `json:"cost_seconds"`
	Evidence    Evidence `json:"evidence"`
}

MetricComparison is one cell of the grid: the p-value one (model, slice, metric) comparison produced, the tier and runtime cost of producing it, and the spine Evidence it came from. Evidence carries the provenance the epic requires (model, tokenizer, engine/backend, seed or deterministic oracle, code revision, tolerance/baseline) plus — on a failing run — the localized first divergence and the scrubbed replay bundle, so a discovery arrives already actionable. EvidenceFromResult (release_gate.go) is the seam that builds it from a RunCase Result.

type MultiplicityCorrection added in v0.44.0 

type MultiplicityCorrection string

MultiplicityCorrection is the closed set of correction policies a family may declare. Each names the error rate it bounds; CorrectionNone names the absence of one and is admitted only so the cost of skipping correction can be MEASURED (see the null simulation in the test file) — a family that declares it is refused a pass. 

const (
	// CorrectionHolm is Holm-Bonferroni step-down: strong family-wise control,
	// uniformly more powerful than plain Bonferroni, no independence assumption.
	CorrectionHolm MultiplicityCorrection = "holm"
	// CorrectionBenjaminiHochberg is BH step-up: false-discovery-rate control for
	// independent or PRDS p-values, the right trade on a wide model x slice grid
	// where insisting on zero false alarms costs every real detection.
	CorrectionBenjaminiHochberg MultiplicityCorrection = "benjamini-hochberg"
	// CorrectionNone applies no correction. It bounds nothing.
	CorrectionNone MultiplicityCorrection = "none"
)

type MultiplicityDecision added in v0.44.0 

type MultiplicityDecision struct {
	Schema      string               `json:"schema"`
	Policy      MultiplicityPolicy   `json:"policy"`
	Bound       string               `json:"bound"`
	Comparisons int                  `json:"comparisons"`
	Tested      int                  `json:"tested"`
	Gated       int                  `json:"gated"`
	Decisions   []ComparisonDecision `json:"decisions"`
	Blocks      []ComparisonDecision `json:"blocks,omitempty"`
	Refusals    []string             `json:"refusals,omitempty"`
	Cost        []TierCost           `json:"cost"`
	Pass        bool                 `json:"pass"`
}

MultiplicityDecision is the machine-readable output of the controller: the declared policy, the bound that policy actually buys, every comparison's decision, the blocking subset ordered strongest-evidence-first, any family-level refusal, and the per-tier cost. Pass is true iff nothing blocked and nothing was refused.

func ControlMultiplicity added in v0.44.0 

func ControlMultiplicity(policy MultiplicityPolicy, family []MetricComparison) (MultiplicityDecision, error)

ControlMultiplicity adjudicates one family of comparisons under one policy. It admits each comparison (provenance, tier, cost, interpretable p-value), applies the declared correction to the PRIMARY sub-family, opens the secondary gate only if the primary family rejected, and folds the result into a decision whose blocking entries lead with the strongest evidence. It is a pure function of (policy, family) — same inputs, same decision — so a multiplicity verdict replays.

type MultiplicityPolicy added in v0.44.0 

type MultiplicityPolicy struct {
	Alpha      float64                `json:"alpha"`
	Correction MultiplicityCorrection `json:"correction"`
	Primary    []string               `json:"primary_metrics"`
}

MultiplicityPolicy is the declaration a family is held to: the error budget, the correction that spends it, and which metrics are PRIMARY. Declaring the primary set is mandatory — the hierarchy is what the secondary gate opens behind, and a family with no primary has nothing to gate on.

func (MultiplicityPolicy) Validate added in v0.44.0 

func (p MultiplicityPolicy) Validate() error

Validate refuses a policy that cannot bound anything: an alpha outside (0, 1), an unknown correction, or an undeclared primary metric set.

type OAISemantics 

type OAISemantics struct{}

OAISemantics is the OpenAI-compatible response-semantics oracle (#4547): it qualifies the response ENVELOPE an OpenAI-compatible serving front end wraps around a decode, not the decode itself. A token stream can be bit-exact against the reference while the response still lies about WHY the decode ended (finish_reason "stop" on a length-truncated completion silently drops the "your answer was cut off" signal), miscounts usage (billing and budget gates key off prompt+completion=total), or ships a message shape a client rejects. Those defects are invisible to token-differential oracles, so this rubric judges the envelope directly.

The engine's response travels as JSON in eng.Text (the additive Trace seam); eng.Tokens remains the witnessed completion token stream the usage block is audited against. Five deterministic checks run, each named by the response field it audits:

  1. message.role — the first choice's message role must be "assistant".
  2. message.content — an assistant message must carry content or tool_calls.
  3. finish_reason — must match the reason derived from ground truth: a message carrying tool_calls must say "tool_calls"; a decode whose trace hit Params.MaxTokens must say "length"; otherwise "stop". A value outside the closed vocabulary ("stop", "length", "tool_calls", "content_filter") is its own violation; "content_filter" is recognized but never expected here — hermetic spine traces model no moderation.
  4. usage arithmetic — prompt_tokens + completion_tokens must equal total_tokens, with no negative counts.
  5. usage/trace agreement — completion_tokens must equal the number of tokens the trace actually witnessed.

Score = passed checks / 5; Pass iff Score >= Rubric.MinScore (default 1: every semantics check must hold). On failure Detail names the FIRST bad field — `finish_reason: got "stop", want "length"` — per the spine's localization contract. An unparseable, choiceless, or messageless response fails closed at score 0: a response that cannot be read has no verifiable semantics.

func (OAISemantics) Judge 

func (OAISemantics) Judge(_ Trace, eng Trace, c QualityCase) Verdict

func (OAISemantics) Kind 

func (OAISemantics) Kind() string

func (OAISemantics) Name 

func (OAISemantics) Name() string

type Observation added in v0.44.0 

type Observation struct {
	Case     QualityCase `json:"case"`
	Evidence Evidence    `json:"evidence"`
}

Observation is ONE case's outcome in ONE run: the case (which carries the model, tokenizer, backend, mode, tier, and cost header) and the evidence it produced (state, provenance, first divergence, scrubbed replay artifact). It is the unit a harness submits; the auto-filer never runs anything itself.

func DemoFilingObservation added in v0.44.0 

func DemoFilingObservation(revision, defect string) Observation

DemoFilingObservation runs the demo case through the REAL spine at the given revision with an optional injected defect ("" = clean, "decode"/"stop"/"report" = planted) and folds the Result into an Observation. The evidence is produced by RunCase rather than hand-written, so the auto-filer is proven against genuine first-divergence evidence and a genuinely scrubbed bundle — the planted-defect-fails / fix-passes witness the epic requires.

type Oracle 

type Oracle interface {
	// Name is the stable identifier a case's Oracles list references.
	Name() string
	// Kind is "differential" (exact/tolerance comparison to a reference) or
	// "rubric" (a scored quality dimension). It drives how explain localizes.
	Kind() string
	// Judge compares eng against ref for case c. It must be a pure function of its
	// inputs so a result replays identically.
	Judge(ref, eng Trace, c QualityCase) Verdict
}

Oracle judges an engine trace against a reference trace for one case and returns a Verdict. Two kinds ship in the spine: a DETERMINISTIC comparator (token-by-token differential) and a RUBRIC scorer (deterministic grounding stand-in). The child cohort registers more oracles against this same interface — logit-parity tolerances (#4523), distribution comparison (#4530), calibrated report judges (#4563) — without editing this file.

func Lookup 

func Lookup(names []string) ([]Oracle, error)

Lookup resolves the oracles a case names, preserving the case's order. It returns an error naming the first unknown oracle so a typo'd or unshipped oracle is refused rather than silently skipped (a skipped oracle is not a pass).

type OracleEvidence 

type OracleEvidence struct {
	Kind     string `json:"kind"`
	Revision string `json:"revision"`
}

OracleEvidence describes deterministic evidence when a case is not seed-pinned.

type PPParity 

type PPParity struct{}

PPParity is the differential oracle for pipeline-parallel execution (#4543): a decode split across stages — including one resumed from a stage-boundary snapshot — must emit exactly the token sequence the single-stage reference emits. Any activation dropped or duplicated in a handoff, or any stage state lost across a boundary resume, is reported as the FIRST divergence, so "the sharded engine reads wrong" localizes to "token 3 was 'cairn' where the reference emitted 'fell'".

func (PPParity) Judge 

func (PPParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (PPParity) Kind 

func (PPParity) Kind() string

func (PPParity) Name 

func (PPParity) Name() string

type PPPipelineRunner 

type PPPipelineRunner struct {
	Label  string
	Resume int
	// contains filtered or unexported fields
}

PPPipelineRunner is the engine path: it decodes with the layers split across two stages, handing the activation across the boundary for every token, and — when Resume names a token in range — interrupting AT the stage boundary of that token: stage one has run, stage two has not, and only what round-trips through the byte snapshot survives. The zero defect is a faithful pipeline; the defect field (set via PPPipelineEngine) injects a boundary bug.

func PPPipelineEngine 

func PPPipelineEngine(resume int, defect string) PPPipelineRunner

PPPipelineEngine returns a staged engine runner that resumes at the stage boundary of token resume (a resume out of [0, MaxTokens) never triggers), with an optional injected defect: "" pipelines and resumes faithfully; "drop-handoff" loses token ppDefectToken's activation in the stage handoff; "dup-handoff" delivers that activation to stage two twice; "resume-state-loss" restores the boundary snapshot without stage two's carried state. These are the deterministic mutants the tests use to prove the parity gate trips at the affected token.

func (PPPipelineRunner) Name 

func (p PPPipelineRunner) Name() string

func (PPPipelineRunner) Run 

func (p PPPipelineRunner) Run(c QualityCase) (Trace, error)

type PairwiseDimension 

type PairwiseDimension struct {
	Name     string   `json:"name"`
	Criteria []string `json:"criteria"`
	Weight   float64  `json:"weight,omitempty"`
}

PairwiseDimension is one rubric dimension scored independently for the baseline and the candidate report. Score = fraction of Criteria present in that side's text. Weight (default 1) sets its pull on the aggregate net margin.

type PairwiseDimensionResult 

type PairwiseDimensionResult struct {
	Name           string  `json:"name"`
	BaselineScore  float64 `json:"baseline_score"`
	CandidateScore float64 `json:"candidate_score"`
	Winner         string  `json:"winner"`
	Rationale      string  `json:"rationale"`
}

PairwiseDimensionResult is the preserved rationale for one scored dimension: each side's score, the winner (ties allowed), and a human rationale naming which criteria each side covered and missed. It is the audit trail that makes a paired verdict reviewable rather than a bare number.

type PairwiseOutcome 

type PairwiseOutcome struct {
	Winner     string                    `json:"winner"`
	Order      string                    `json:"order"`
	NetMargin  float64                   `json:"net_margin"`
	Dimensions []PairwiseDimensionResult `json:"dimensions"`
}

PairwiseOutcome is the full result of one baseline-vs-candidate evaluation: the overall winner, the presentation order the sides were judged in (the randomized-side-order evidence — the outcome is invariant to it), the weighted net margin (candidate minus baseline, positive favors the candidate), and every per-dimension rationale.

func EvaluatePairwise 

func EvaluatePairwise(baseline, candidate string, spec PairwiseSpec, seed int64) PairwiseOutcome

EvaluatePairwise scores a candidate report against a baseline report across the spec's rubric dimensions and folds the per-dimension comparisons into an overall paired outcome. It is a pure function of its inputs — the same (baseline, candidate, spec, seed) always yields the same outcome — so a paired verdict replays. seed selects only the presentation order (recorded, not scored): the outcome is invariant to it.

type PairwiseSpec 

type PairwiseSpec struct {
	Dimensions []PairwiseDimension `json:"dimensions"`
	Expect     string              `json:"expect,omitempty"`
	TieMargin  float64             `json:"tie_margin,omitempty"`
}

PairwiseSpec is the prompt-carried configuration of a baseline-vs-candidate evaluation: the rubric dimensions to score, the expected winner the case asserts, and the tie margin below which a score gap is called a tie. A zero TieMargin means an exact tie (equal scores) is the only tie.

type PenaltyOrdering 

type PenaltyOrdering struct{}

PenaltyOrdering is the penalty math + ordering differential oracle (#4527).

func (PenaltyOrdering) Judge 

func (PenaltyOrdering) Judge(ref, eng Trace, c QualityCase) Verdict

func (PenaltyOrdering) Kind 

func (PenaltyOrdering) Kind() string

func (PenaltyOrdering) Name 

func (PenaltyOrdering) Name() string

type PointState 

type PointState string

PointState is the closed classification of one probed point. Good and Bad are the two monotone rungs a bisect narrows between; Indeterminate is the fail-closed third rung — missing, stale, mis-attributed, or inconclusive evidence that can be neither trusted as good nor localized as the cause. Its existence is the point: "missing or inconclusive evidence is never pass" (#4583 acceptance) applied to bisect means such a point halts the search instead of being guessed past. 

const (
	PointGood          PointState = "good"          // releasing evidence at this revision
	PointBad           PointState = "bad"           // a localized divergence at this revision
	PointIndeterminate PointState = "indeterminate" // unclassifiable — never good, never a fabricated cause
)

type PostSelectionDecision added in v0.44.0 

type PostSelectionDecision struct {
	Schema        string             `json:"schema"`
	Certification SweepCertification `json:"certification"`
	// FamilySize is the number of arms actually handed over, which is what the
	// adjustment is computed over once the declaration has been reconciled with it.
	FamilySize int          `json:"family_size"`
	Winner     *ArmDecision `json:"winner,omitempty"`
	// Unadjusted is the retained arm's raw p-value: confidence at the winning point,
	// ignoring that the point was chosen by looking.
	Unadjusted float64 `json:"unadjusted_p"`
	// Adjusted is that p-value priced over the complete search family. On a refusal
	// it is pinned at 1 — no family was established, so no bound was bought.
	Adjusted float64 `json:"selection_adjusted_p"`
	// SelectionPrice is Adjusted - Unadjusted: what the search breadth cost, in
	// p-value, stated as its own number so it can be tracked over time.
	SelectionPrice      float64 `json:"selection_price"`
	UnadjustedCertifies bool    `json:"unadjusted_certifies"`
	Certified           bool    `json:"certified"`
	// ManufacturedByBreadth is the headline finding: the sweep's unadjusted result
	// clears alpha and its priced result does not, so the passing claim was produced
	// by how widely it searched rather than by the improvement it found. It is only
	// ever set on a decision that was priced — a refusal is a different failure and
	// is not laundered into this one.
	ManufacturedByBreadth bool                   `json:"manufactured_by_breadth"`
	Bound                 string                 `json:"bound"`
	Arms                  []ArmDecision          `json:"arms"`
	Refusals              []PostSelectionRefusal `json:"refusals,omitempty"`
}

PostSelectionDecision is the machine-readable output of the controller: the declaration, the family it was checked against, every arm, and — the point of the layer — the unadjusted and selection-adjusted results side by side with the price of the search between them.

Certified is true only when nothing was refused AND the SELECTION-ADJUSTED p-value clears alpha. UnadjustedCertifies records what the naive read would have concluded, so the two are never collapsed into one number.

func PriceSearch added in v0.44.0 

func PriceSearch(cert SweepCertification, family []SearchArm) PostSelectionDecision

PriceSearch adjudicates one searched sweep against its declaration. It admits every arm, reconciles the declared family size with the arms on the record, checks that the retained arm is the one the declared rule would have picked, and prices the winner's p-value over the complete family.

It returns no error: every way the contract can fail is a typed refusal ON the decision, because the receipt has to record refusals as faithfully as it records passes. A caller that ignores Refusals entirely and reads only Certified still fails closed — Certified is false whenever anything was refused.

It is a pure function of (certification, family): same inputs, same decision, so a certification replays.

type PostSelectionRefusal added in v0.44.0 

type PostSelectionRefusal struct {
	Code   PostSelectionRefusalCode `json:"code"`
	Detail string                   `json:"detail"`
}

PostSelectionRefusal is one typed reason the contract withheld a conclusion. Detail localizes it to the specific arm or field so the refusal is actionable.

type PostSelectionRefusalCode added in v0.44.0 

type PostSelectionRefusalCode string

PostSelectionRefusalCode is the closed vocabulary of typed refusals the contract uses. A structured code is what lets a consumer branch on WHY a certification was withheld instead of pattern-matching prose. 

const (
	// RefuseCertificationInvalid: the declaration itself cannot bound anything —
	// an alpha outside (0, 1), an unknown rule or adjustment, or no retained arm
	// named.
	RefuseCertificationInvalid PostSelectionRefusalCode = "certification_invalid"
	// RefuseFamilyUndeclared: no search family was handed over, or its size was
	// never declared. The winner cannot be priced against a family that does not
	// exist on the record.
	RefuseFamilyUndeclared PostSelectionRefusalCode = "search_family_undeclared"
	// RefuseFamilyIncomplete: the declared family size disagrees with the arms
	// handed over. Fewer arms than declared means evidence was evaluated and not
	// reported (the file drawer) and the selection cannot be audited; more arms
	// than declared means the declaration understates the breadth actually searched
	// and would underprice it.
	RefuseFamilyIncomplete PostSelectionRefusalCode = "search_family_incomplete"
	// RefuseWinnerNotInFamily: the retained arm is not one of the arms handed over,
	// so the family the price is computed over is not the family it was drawn from.
	RefuseWinnerNotInFamily PostSelectionRefusalCode = "winner_not_in_family"
	// RefuseWinnerNotExtremal: the retained arm is not the arm the declared rule
	// would have picked. The selection rule on the record is then not the rule that
	// was used, and the adjustment derived from it does not apply.
	RefuseWinnerNotExtremal PostSelectionRefusalCode = "winner_not_extremal"
	// RefuseArmInadmissible: an arm's evidence cannot be interpreted (missing
	// provenance, unproduced evidence, a p-value outside [0, 1]). An uninterpretable
	// arm leaves the true breadth of the search unknown, so the family is refused
	// rather than silently shrunk to the arms that happened to parse.
	RefuseArmInadmissible PostSelectionRefusalCode = "arm_inadmissible"
	// RefuseAdjustmentUndeclared: the sweep searched more than one point and
	// declared no adjustment, so the reported confidence honors no bound at all.
	RefuseAdjustmentUndeclared PostSelectionRefusalCode = "selection_adjustment_undeclared"
)

type PrefillChunkedRunner 

type PrefillChunkedRunner struct {
	Label     string
	ChunkSize int
	// contains filtered or unexported fields
}

PrefillChunkedRunner is the engine path: it splits the prompt tokens into chunks of ChunkSize and prefill-processes them sequentially, folding the carried state chunk by chunk before decoding. With no defect it is a faithful engine (carried state makes it identical to monolithic at ANY split point); the defect field (set via PrefillEngine) injects the boundary bug.

func PrefillEngine 

func PrefillEngine(chunkSize int, defect string) PrefillChunkedRunner

PrefillEngine returns a chunked engine runner that splits the prompt at chunkSize with an optional injected defect: "" folds carried state faithfully across chunk boundaries (parity holds at every split point); "state-reset" resets the accumulated state at each chunk boundary so any multi-chunk split diverges from the monolithic decode. This is the deterministic mutant source the tests use to prove the parity gate trips.

func (PrefillChunkedRunner) Name 

func (r PrefillChunkedRunner) Name() string

func (PrefillChunkedRunner) Run 

func (r PrefillChunkedRunner) Run(c QualityCase) (Trace, error)

type PrefillMonolithicRunner 

type PrefillMonolithicRunner struct{}

PrefillMonolithicRunner is the reference path for #4536: it prefill-processes the whole prompt in one pass and decodes Params.MaxTokens generated tokens from the accumulated state. It is the "what correct looks like" side of the parity differential.

func (PrefillMonolithicRunner) Name 

func (PrefillMonolithicRunner) Name() string

func (PrefillMonolithicRunner) Run 

func (r PrefillMonolithicRunner) Run(c QualityCase) (Trace, error)

type PrefillParity 

type PrefillParity struct{}

PrefillParity is the differential oracle for #4536: the chunked engine's generated token stream must equal the monolithic reference stream exactly. Any mismatch — a reset boundary, a mis-carried accumulator, a truncated decode — is reported as the FIRST divergent generated token, so "chunked prefill broke generation" localizes to "output token 2 was 'drift' where monolithic emitted 'ember'".

func (PrefillParity) Judge 

func (PrefillParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (PrefillParity) Kind 

func (PrefillParity) Kind() string

func (PrefillParity) Name 

func (PrefillParity) Name() string

type Probe 

type Probe func(BisectPoint) Evidence

Probe produces the qualification Evidence for one lineage point — the expensive step (check out the revision under the engine/config, run the case, submit the Evidence) the bisect calls lazily at each visited point. It is the seam a real bisect wires a git-checkout+RunCase driver into; tests wire a deterministic planted-defect probe. The bisect is a pure function of (lineage, Probe): a deterministic Probe yields a deterministic verdict, so a bisect result replays — the same conditional purity RunCase has on its Runner adapters.

func DemoBisectProbe 

func DemoBisectProbe(badFrom int) Probe

DemoBisectProbe returns a deterministic probe that models a regression planted at revision index badFrom: every revision at or after badFrom decodes with the "decode" defect (the greedy oracle fails at token 1), every earlier revision decodes cleanly, and badFrom < 0 models a fully-fixed lineage where no revision is bad. It runs the REAL spine (RunCase over DemoCase) at each point and folds the Result into Evidence, so the bisect is proven against genuine first-divergence evidence rather than a hand-written verdict — the "planted defect fails, fix passes, independently replayed" witness the epic requires.

type Provenance 

type Provenance struct {
	CaseID    string         `json:"case_id"`
	Params    SamplingParams `json:"params"`
	Reference Trace          `json:"reference"`
	Engine    Trace          `json:"engine"`
	// Requests is what each side did with those params: which normalized request
	// fields it could not honor and how the request it actually executed differed
	// (#4518). Without it a passing run only asserts that two traces agree, not
	// that they answer the same question.
	Requests RequestRecord `json:"requests"`
}

Provenance is the captured request/config/token/logit/output evidence for a run (spine contract item 2). It is what makes a failure localizable: the exact params both paths ran under and the full reference vs engine traces.

type QualityCase 

type QualityCase struct {
	Schema  string         `json:"schema"`
	ID      string         `json:"id"`
	Version int            `json:"version"`
	Prompt  string         `json:"prompt"`
	Params  SamplingParams `json:"params"`
	// Reference is the golden trace the engine path is compared against. For a
	// deterministic (temperature-zero / greedy) case this is an exact oracle; a
	// stochastic case carries a distribution instead (added by the sampling
	// children #4529–#4530 without touching this struct).
	Reference Trace `json:"reference"`
	// Oracles names the comparators/scorers to apply, by Oracle.Name(). An empty
	// list is a case authored without any judge — RunCase reports it rather than
	// silently passing (a case that checks nothing is not a green case).
	Oracles []string `json:"oracles"`
	// Rubric configures the rubric scorer(s) for this case (grounding phrases,
	// forbidden claims, threshold). Optional; empty means no rubric dimension.
	Rubric RubricSpec `json:"rubric,omitempty"`
	// Metadata pins the execution and baseline provenance required by canonical
	// v1 corpus fixtures. LoadCase rejects a fixture when any required lineage,
	// determinism, tier, or resource-cost field is absent.
	Metadata CaseMetadata `json:"metadata,omitempty"`
	Tags     []string     `json:"tags,omitempty"`
}

QualityCase is one versioned, replayable unit of quality evidence: a prompt, the sampling configuration it must be decoded under, the reference expectation it is judged against, and the named oracles to apply. It carries everything a runner needs to reproduce the run — nothing about a case depends on wall-clock or ambient state.

func BatchInvarianceCase 

func BatchInvarianceCase() QualityCase

BatchInvarianceCase builds the demo sweep: the target decoded alone, then leading, middle, and trailing a batch of three. The reference is the target's ALONE decode; the case asserts every batched placement reproduces it exactly.

func DemoCase 

func DemoCase() QualityCase

DemoCase is the built-in spine case: a tiny greedy executive-report decode with a grounded reference and a rubric requiring the two material claims. `fak quality run` uses it so the spine is runnable end-to-end with no corpus on disk, and the epic Witness ("intentionally injected decode and report-quality defects each trip the expected gate") exercises it via DemoEngine.

func DemoFilingCase added in v0.44.0 

func DemoFilingCase(revision string) QualityCase

DemoFilingCase stamps the spine demo case with the canonical routing header and baseline provenance a filed finding must document (#4584 acceptance 2 and 4): model, tokenizer, engine/backend and mode, deterministic oracle, code revision, tolerance and baseline, an explicit tier, and the runtime/resource cost. It is the hermetic fixture the witness folds, and the shape a real corpus case has.

func DistributionCase 

func DistributionCase(id string, vocab []string, probs []float64, draws int, seed int64) QualityCase

DistributionCase builds a stochastic quality case whose reference is a DISTRIBUTION rather than a single golden trace: the vocabulary goes in Reference.Tokens and the aligned target probabilities as the one row Reference.Logits[0] — no change to the QualityCase struct. MaxTokens is the number of draws the engine runner must make and Seed pins its PRNG, so the case replays identically.

func DtypeDeltaCase 

func DtypeDeltaCase() QualityCase

DtypeDeltaCase builds the dtype-parity fixture: a temperature-zero decode whose reference trace carries the FP32 logits every dtype lane is judged against.

func KVEvictionCase 

func KVEvictionCase(seed int64) QualityCase

KVEvictionCase builds the eviction-parity case: a greedy (temperature-zero) decode long enough that the engine's sliding window MUST evict and recompute (MaxTokens > kvEvictionWindow+1), with the reference trace produced by the never-evicted full-KV decode under the same seed.

func LoadCase 

func LoadCase(data []byte) (QualityCase, error)

LoadCase decodes one canonical v1 fixture, refusing unknown fields, trailing documents, malformed provenance, and fixtures that cannot prove determinism.

func LogprobParityCase 

func LogprobParityCase() QualityCase

LogprobParityCase builds the deterministic logprob-parity case: a fixed multi-token prompt, a temperature-zero generation budget, and a reference trace carrying one normalized logprob row per prompt AND generated token, produced by the same scorer the faithful engine runs.

func PPParityCase 

func PPParityCase(seed int64, steps int) QualityCase

PPParityCase builds a pipeline-parity case: a deterministic layered decode of steps tokens under seed, whose reference trace is the single-stage run. The case says nothing about HOW the engine is staged or WHERE it resumes — a correct pipeline split and a correct boundary resume must both be invisible in the output.

func PenaltyCase 

func PenaltyCase(id string, vocab []string, raw []float64, counts map[string]int, rep, presence, freq float64) QualityCase

PenaltyCase builds a penalty-ordering quality case: the raw logits row and vocabulary ride in the standard Reference slots, and the penalty strengths + history counts are serialized into the Prompt so the case remains pure, hermetic, replayable data. counts maps a vocab token to how many times it already appeared in the generation history; tokens absent (or with count 0) are unpenalized.

func PrefillParityCase 

func PrefillParityCase() QualityCase

PrefillParityCase builds the deterministic parity case: a fixed multi-token prompt, a temperature-zero decode budget, and a reference trace produced by the monolithic prefill path itself. Chunk geometry deliberately does NOT appear in the case — parity must hold for EVERY split point, so the split is engine configuration, not case data.

func QuantBudgetCase 

func QuantBudgetCase(id string, budget float64) QualityCase

QuantBudgetCase builds a hermetic quantization-budget case: a deterministic quantBudgetSteps-token full-precision reference stream, judged only by the quantization-budget oracle under the declared budget (0 declares none, so the oracle applies its default).

func ResumeParityCase 

func ResumeParityCase(seed int64, steps int) QualityCase

ResumeParityCase builds a resume-parity case: a deterministic stateful decode of steps tokens under seed, whose reference trace is the uninterrupted run. The case itself says nothing about WHERE the engine is interrupted — a correct resume must be invisible in the output for any k.

func SchedulerParityCase 

func SchedulerParityCase() QualityCase

SchedulerParityCase builds the scheduler-parity case: the demo workload as the Prompt, and a reference trace produced by a faithful fcfs run — the "one policy" every other policy's outputs are compared against. The case says nothing about WHICH policy the engine runs; parity must hold for all of them.

func SeedReplayCase 

func SeedReplayCase(seed int64) QualityCase

SeedReplayCase builds a stochastic replay case pinned to seed: temperature above zero (this is sampling, not greedy decode), a fixed step budget, and a reference trace produced by the deterministic sampler itself under that seed. Replay is SEED-SCOPED: the case asserts that the SAME seed reproduces the same sequence, not that generation is globally constant — a different seed may (and for this vocab does) produce a different sequence.

func SpecDecodeCase 

func SpecDecodeCase(seed int64) QualityCase

SpecDecodeCase builds the speculative-parity case: a greedy (temperature-zero) decode pinned to seed, whose reference trace is target-only decode under that seed. MaxTokens spans several draft blocks and multiple drift steps, so both the accept path and the reject-fallback path are exercised.

func TZCase 

func TZCase() QualityCase

TZCase builds the temperature-zero surface-parity case: temperature pinned to zero, a nonzero seed ON PURPOSE (a faithful temp-0 decode must be identical regardless of seed — the seed exists so a defective path has an RNG to leak from), and a greedy reference produced by the shared decode path.

func TokenizerParityCase 

func TokenizerParityCase() QualityCase

TokenizerParityCase builds the deterministic parity case: a two-turn system+user conversation serialized as JSON into Prompt, and the pinned-correct id sequence — rendered once by the pure template — as the reference trace. The reference IS the pin: any engine rendering that departs from it, at any id, special or content, is a template-parity defect.

func TopKTopPCase 

func TopKTopPCase(id string, vocab []string, logits []float64, topK int, topP float64) QualityCase

TopKTopPCase builds a boundary case for this oracle: the vocabulary goes in Reference.Tokens, the aligned logits row as the single row Reference.Logits[0], and the k/p under test in Params — no change to the QualityCase struct. The engine runner under test must emit the candidate set it kept, in ranking order, as its Trace.Tokens.

func TpParityCase 

func TpParityCase() QualityCase

TpParityCase builds the deterministic tensor-parallel parity case: a temperature-zero decode budget and a reference trace produced by the TP=1 (single shard) path itself. The TP degree deliberately does NOT appear in the case — parity must hold for EVERY degree, so the degree is engine configuration, not case data.

func (QualityCase) Valid 

func (c QualityCase) Valid() (bool, string)

Valid reports whether the case envelope is well-formed enough to run. It is the admission gate: a case with the wrong schema, no ID, or no oracles is refused with a reason rather than run to a misleading green.

func (QualityCase) ValidateCanonical 

func (c QualityCase) ValidateCanonical() error

ValidateCanonical enforces the complete v1 corpus contract. Valid remains the compatibility admission gate for cases constructed by older in-package helpers; persisted fixtures enter through LoadCase and always receive this stricter check.

type QuantBudget 

type QuantBudget struct{}

QuantBudget is the quantization-budget oracle (#4540): a statistical differential gate that scores the engine's token-agreement rate against the reference and passes iff it meets the case's declared budget. The budget is declared per case in Rubric.MinScore (a fraction in (0, 1]); a case that declares none gets quantDefaultAgreementBudget. Score is the measured agreement rate; on failure FirstDivergence pins the first disagreeing position and Detail reports measured agreement vs budget.

func (QuantBudget) Judge 

func (QuantBudget) Judge(ref, eng Trace, c QualityCase) Verdict

func (QuantBudget) Kind 

func (QuantBudget) Kind() string

func (QuantBudget) Name 

func (QuantBudget) Name() string

type QuantBudgetEngine 

type QuantBudgetEngine struct {
	Label         string
	MismatchEvery int
}

QuantBudgetEngine is the engine-side adapter modeling a quantized decode of the case's reference: it replays the reference stream with every MismatchEvery-th position flipped by quantBudgetPerturb (rounding error). MismatchEvery <= 0 is a bit-faithful quantization; a large value models a healthy build whose drift sits inside the budget; a small value models a defective quantization whose drift exceeds it. This is the deterministic mutant source the tests use to prove the gate trips.

func (QuantBudgetEngine) Name 

func (e QuantBudgetEngine) Name() string

func (QuantBudgetEngine) Run 

func (e QuantBudgetEngine) Run(c QualityCase) (Trace, error)

type ReferenceRunner 

type ReferenceRunner struct{}

ReferenceRunner is the golden path: it returns the case's declared reference trace verbatim, stamped with its own runner name. It is the baseline every differential oracle names — the "what the case says correct looks like" side.

func (ReferenceRunner) EffectiveRequest added in v0.44.0 

func (ReferenceRunner) EffectiveRequest(c QualityCase) EffectiveRequest

EffectiveRequest makes ReferenceRunner the package's one CONCRETE RequestAdapter (#4518). The reference is the baseline every other verdict is measured against, so leaving it merely ASSUMED faithful is the weakest link in the record: a run could report an audited engine against an unaudited golden path and still call itself evidence. The declaration is exact rather than generous — the case's own prompt and params, with nothing unsupported — because a reference trace is BY DEFINITION the answer to the case's own request. Declaring it costs nothing on a faithful run (requestFidelity finds no drop and no diff, so no verdict is appended) and upgrades every result's reference record from Declared:false to an actual measurement.

func (ReferenceRunner) Name 

func (ReferenceRunner) Name() string

func (ReferenceRunner) Run 

func (r ReferenceRunner) Run(c QualityCase) (Trace, error)

type RegressionKey added in v0.44.0 

type RegressionKey struct {
	CaseID   string `json:"case_id"`
	Model    string `json:"model"`
	Backend  string `json:"backend"`
	Mode     string `json:"mode"`
	Metric   string `json:"metric"`
	FirstBad string `json:"first_bad"`
}

RegressionKey is the identity an auto-filed finding is keyed by — the #4584 scope vocabulary exactly: which case, on which model, which backend, which engine mode, which metric regressed, and where it first went bad. Two findings with equal keys are the same defect observed again; two findings with different keys are different defects and get different issues.

FirstBad is the first ACTIONABLE divergence coordinate, not a summary: the token index the streams first disagreed at when there is one, else the serving stage the bundle's own evidence attributes the failure to, else the failing kind. Keying on it means a regression that moves to a different token is a different issue — which is correct, because it is a different defect to fix.

func (RegressionKey) Marker added in v0.44.0 

func (k RegressionKey) Marker() string

Marker is the deduplication handle a driver matches on in the tracker of record: an HTML comment carrying the key, stable across runs and safe to embed in an issue body. It is built per-FIELD and is INJECTIVE — distinct keys always produce distinct markers — which is the whole contract: a lossy marker would silently fold two different defects into one issue, the exact mistake deduplication is supposed to prevent. Field values are percent-escaped, so the joining "/" is unambiguously a separator even for a case id that contains one (the nightly matrix mints exactly those), and no value can smuggle in a "<", ">", newline, or "--" that would close the comment early.

func (RegressionKey) String added in v0.44.0 

func (k RegressionKey) String() string

String renders the key as its slash-joined coordinates, empty axes shown as "-" so a missing axis is visible rather than swallowed.

type ReleaseBlock 

type ReleaseBlock struct {
	CaseID          string         `json:"case_id"`
	Tier            Tier           `json:"tier"`
	Kind            GateKind       `json:"kind"`
	State           EvidenceState  `json:"state"`
	Reason          string         `json:"reason"`
	FirstDivergence *Divergence    `json:"first_divergence,omitempty"`
	Replay          *FailureBundle `json:"replay,omitempty"`
}

ReleaseBlock is one required gate that did not release, with the reason it blocked and — where the evidence carried it — the first actionable divergence and scrubbed replay artifact. The gate emits these first-actionable-first so an operator reads the earliest blocking gate, not an averaged summary.

type ReleaseDecision 

type ReleaseDecision struct {
	Schema   string         `json:"schema"`
	Revision string         `json:"revision"`
	Released bool           `json:"released"`
	Blocks   []ReleaseBlock `json:"blocks,omitempty"`
	Passed   []string       `json:"passed,omitempty"`
}

ReleaseDecision is the machine-readable output of the release gate: whether the revision may ship, every gate that blocked it, and the case IDs that qualified. Released is true iff every required gate has fresh, passing, fully-attributed evidence — the fail-closed contract of #4578.

func QualifyRelease 

func QualifyRelease(rev string, required []RequiredGate, evidence []Evidence) ReleaseDecision

QualifyRelease adjudicates whether revision `rev` may ship given the required gates and the produced evidence. A required gate blocks when its evidence is missing, lacks complete provenance (inconclusive), was produced at a different revision (stale), or did not pass. Blocks preserve the required-gate order, so Blocks[0] is the first actionable divergence. It is a pure function of its inputs — same inputs, same decision — so a release verdict replays.

type ReplaySignature added in v0.44.0 

type ReplaySignature struct {
	FailingOracle   string      `json:"failing_oracle"`
	FailingKind     string      `json:"failing_kind"`
	FirstDivergence *Divergence `json:"first_divergence,omitempty"`
}

ReplaySignature is the identity of a failure: which oracle failed, of which kind, and where the token streams first diverged. It is what a replay must reproduce for the bundle to be considered replay-complete — the same localizing evidence the epic exists to produce, compared rather than narrated.

func (ReplaySignature) String added in v0.44.0 

func (s ReplaySignature) String() string

String renders a signature as the one line an operator reads to compare an expected failure against a replayed one.

type ReplayVerdict added in v0.44.0 

type ReplayVerdict struct {
	Schema       string           `json:"schema"`
	CaseID       string           `json:"case_id"`
	Reproduced   bool             `json:"reproduced"`
	Inconclusive bool             `json:"inconclusive"`
	Reason       string           `json:"reason"`
	Expected     ReplaySignature  `json:"expected"`
	Observed     *ReplaySignature `json:"observed,omitempty"`
	// Result is the freshly replayed run, present whenever the replay actually
	// executed. It carries its own manifest and provenance, so a replay is itself
	// an auditable artifact rather than a bare boolean.
	Result *Result `json:"result,omitempty"`
}

ReplayVerdict is the machine-readable outcome of replaying one failure bundle. Reproduced is the only green state; Inconclusive marks a bundle that could not be replayed at all (as opposed to one that replayed to a different or absent failure), so a CI consumer can tell "this artifact is broken" from "this artifact no longer describes the defect it claims".

func Replay added in v0.44.0 

func Replay(b FailureBundle) ReplayVerdict

Replay reproduces a recorded failure from its bundle and nothing else (#4515). It re-runs the embedded case through the bundle's own captured reference and engine traces, using the same RunCase orchestrator that emitted the bundle, and compares the failure it observes against the failure the bundle recorded. It never returns an error: every outcome — replay-incomplete, replayed-clean, replayed-different, reproduced — is a verdict a gate can route on, and only reproduction is green.

type ReportArithmetic 

type ReportArithmetic struct{}

ReportArithmetic is the numeric-claim validator for executive reports (#4557): a rubric oracle that parses the figures a report ASSERTS — "12% week over week", "38 of 40", a trend word — and checks each against the period's ground-truth numbers instead of trusting the report's own prose. It is the arithmetic-layer mirror of the spine's increased-vs-decreased decode defect: a report can be perfectly fluent and still state a percentage that does not follow from prior/current, a trend word that contradicts the delta's sign, a denominator that is not the period's, or a percentage where a raw count belongs. Each of those fails here with a Detail pinpointing the first bad claim (the claim text plus expected vs stated), so "the numbers looked off" localizes to one checkable assertion.

Ground truth travels IN the case, as a small "key: value" block in Reference.Text (see ArithmeticGroundTruth): 

prior: 100
current: 112
denominator: 40

Checks applied to every parsed claim, in report order:

  • Percentage consistency: a stated "P%" must match round((current-prior)/prior*100) within a tolerance. A trend word next to the figure signs it ("decreased 12%" states -12); a bare percentage is compared by magnitude.
  • Trend-word consistency: "increased" requires current > prior, "decreased" requires current < prior, "flat" requires equality.
  • Denominator/units: an "N of M" claim must have N <= M and M equal to the ground-truth denominator; a "%" on a ratio operand, or a week-over-week change stated as a raw count instead of a percentage, is a unit mismatch.

The oracle passes iff all parsed claims are consistent; a report with no parseable numeric claims passes vacuously (the grounding rubric, not this oracle, owns "the required figure is missing"). Numbers are read as plain decimals — grouped figures like "1,200" are out of scope for this layer.

func (ReportArithmetic) Judge 

func (ReportArithmetic) Judge(ref, eng Trace, c QualityCase) Verdict

Judge parses the engine report's numeric claims and validates each against the ground-truth figures carried in the reference text. Score is the fraction of claims found consistent; on failure Detail pinpoints the FIRST inconsistent claim in report order.

func (ReportArithmetic) Kind 

func (ReportArithmetic) Kind() string

func (ReportArithmetic) Name 

func (ReportArithmetic) Name() string

type RequestAdapter added in v0.44.0 

type RequestAdapter interface {
	Runner
	EffectiveRequest(c QualityCase) EffectiveRequest
}

RequestAdapter is the optional half of the reference-runner adapter seam (#4518). A runner that implements it declares what it really ran, so the harness can prove both sides were handed the same request instead of assuming it. A runner that does NOT implement it asserts faithfulness by omission — RequestFidelity.Declared records which of the two a result rests on, so an audited faithful run is distinguishable from an unaudited one.

type RequestFidelity added in v0.44.0 

type RequestFidelity struct {
	Runner string `json:"runner"`
	// Declared reports whether this runner implements RequestAdapter. False
	// means the fidelity below is an assumption, not a measurement.
	Declared bool `json:"declared"`
	// Unsupported is every field the runner claims it cannot honor, in canonical
	// order — the capability claim, recorded whether or not this case exercises it.
	Unsupported []string `json:"unsupported,omitempty"`
	// Dropped is the subset of Unsupported that THIS case actually specifies:
	// the fields the run silently threw away.
	Dropped []string `json:"dropped,omitempty"`
	// Diff is the request the runner executed minus the request the case
	// declared, excluding fields already named in Dropped.
	Diff []FieldDelta `json:"diff,omitempty"`
}

RequestFidelity is the harness-derived record of how faithfully ONE runner executed the case's normalized request. Dropped and Diff are the two ways a request can be lost in translation: a field the runner silently ignores, and a field it substituted a different value for.

func (RequestFidelity) Faithful added in v0.44.0 

func (f RequestFidelity) Faithful() bool

Faithful reports whether this runner executed the case's normalized request without loss. A runner that declares an unsupported field the case never set is still faithful for that case — nothing was translated away.

type RequestRecord added in v0.44.0 

type RequestRecord struct {
	Reference RequestFidelity `json:"reference"`
	Engine    RequestFidelity `json:"engine"`
}

RequestRecord pairs both sides' fidelity records for one run. It is the request-level half of a result's provenance: the evidence that the comparison underneath it was between like and like.

func (RequestRecord) Faithful added in v0.44.0 

func (r RequestRecord) Faithful() bool

Faithful reports whether BOTH sides ran the case's normalized request.

type RequiredGate 

type RequiredGate struct {
	CaseID      string   `json:"case_id"`
	Tier        Tier     `json:"tier"`
	Kind        GateKind `json:"kind"`
	CostSeconds float64  `json:"cost_seconds"`
}

RequiredGate is one (case, tier, kind) the release MUST have fresh, passing evidence for before it ships. CostSeconds documents the runtime/resource cost of producing the evidence (#4578: assign a tier and document cost) so an operator can see what a release qualification costs per tier.

type Result 

type Result struct {
	Schema        string         `json:"schema"`
	CaseID        string         `json:"case_id"`
	Pass          bool           `json:"pass"`
	Verdicts      []Verdict      `json:"verdicts"`
	Provenance    Provenance     `json:"provenance"`
	Manifest      RunManifest    `json:"manifest"`
	FailureBundle *FailureBundle `json:"failure_bundle,omitempty"`
}

Result is the stable machine-readable outcome of one quality run (#4519): a pass/fail verdict, every oracle's verdict, the captured provenance, the replay manifest, and — iff the run failed — a portable failure bundle. Encoding it as JSON is the CI contract: a PR gate reads Pass, a human reads the bundle.

func RunCase 

func RunCase(c QualityCase, ref, eng Runner, oracles []Oracle) (Result, error)

RunCase is the spine orchestrator (contract items 1–4): it runs the case through the reference and engine runners, captures provenance, applies every named oracle, folds the verdicts into a single pass/fail, and — on failure — attaches a replayable bundle localized to the first failing oracle. It is a pure function of (case, runners, oracles): same inputs, same Result.

type ResumeParity 

type ResumeParity struct{}

ResumeParity is the differential oracle for retry/mid-generation resume (#4538): an interrupted-then-resumed generation must emit exactly the token sequence the uninterrupted reference emits. Any state lost, dropped, or duplicated across the interrupt/restore boundary is reported as the FIRST divergence — the resume boundary is the usual suspect, so "the retry came back different" localizes to "token 3 was 'dune' where the reference emitted 'grove'".

func (ResumeParity) Judge 

func (ResumeParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (ResumeParity) Kind 

func (ResumeParity) Kind() string

func (ResumeParity) Name 

func (ResumeParity) Name() string

type ResumeRunner 

type ResumeRunner struct {
	Label     string
	Interrupt int
	// contains filtered or unexported fields
}

ResumeRunner is the engine path: it decodes k steps, snapshots the live state to bytes (the interrupt), restores a FRESH state value from those bytes, and resumes to completion. The zero defect is a faithful resume; the defect field (set via ResumeEngine) injects a state-loss bug at the interrupt/restore boundary.

func ResumeEngine 

func ResumeEngine(k int, defect string) ResumeRunner

ResumeEngine returns a resuming engine runner interrupted at step k with an optional injected defect: "" resumes faithfully from the byte snapshot; "reseed" loses the restored state and restarts generation from scratch; "drop-boundary" loses the boundary token's emission; "dup-boundary" re-emits the last pre-snapshot token. These are the deterministic mutants the tests use to prove the resume gate trips at the boundary.

func (ResumeRunner) Name 

func (r ResumeRunner) Name() string

func (ResumeRunner) Run 

func (r ResumeRunner) Run(c QualityCase) (Trace, error)

type ResumeUninterruptedRunner 

type ResumeUninterruptedRunner struct{}

ResumeUninterruptedRunner is the reference path: it decodes the case's full Params.MaxTokens steps in one uninterrupted run under Params.Seed. It never snapshots or restores, so its trace is what generation looks like when nothing is ever interrupted.

func (ResumeUninterruptedRunner) Name 

func (ResumeUninterruptedRunner) Name() string

func (ResumeUninterruptedRunner) Run 

func (r ResumeUninterruptedRunner) Run(c QualityCase) (Trace, error)

type Revision 

type Revision struct {
	Name     string `json:"name"`
	Revision string `json:"revision"`
}

Revision identifies immutable model, tokenizer, or code/module content.

type RubricSpec 

type RubricSpec struct {
	Required  []string `json:"required,omitempty"`
	Forbidden []string `json:"forbidden,omitempty"`
	MinScore  float64  `json:"min_score,omitempty"`
}

RubricSpec is the deterministic rubric configuration for a case: phrases the grounded output must contain, claims it must not contain, and the minimum fraction of required phrases to pass. It is intentionally a deterministic stand-in the report-quality children (#4550–#4565) replace with calibrated judges — but even the stand-in makes a "reads fine" report fail when it drops a material, required claim.

func CitationRubric 

func CitationRubric(evidence []string, minScore float64) RubricSpec

CitationRubric is the helper constructor for a case judged by citation-validity: it carries the allowed evidence entries in Rubric.Required — each entry an evidence id, optionally followed by ':' or whitespace and its link/description — and the pass threshold in MinScore (0 means the default: no dangling citation). Blank entries are dropped.

type RunManifest 

type RunManifest struct {
	Schema          string   `json:"schema"`
	CaseID          string   `json:"case_id"`
	CaseVersion     int      `json:"case_version"`
	ReferenceRunner string   `json:"reference_runner"`
	EngineRunner    string   `json:"engine_runner"`
	Oracles         []string `json:"oracles"`
}

RunManifest is the replay-complete record of HOW a result was produced: the case identity and version, which reference/engine runners ran, and which oracles were applied. It deliberately carries no wall-clock or host state — replay identity must not depend on when or where a run happened (#4514). A caller that wants a timestamp stamps it outside, onto the enclosing artifact.

type Runner 

type Runner interface {
	Name() string
	Run(c QualityCase) (Trace, error)
}

Runner is the reference-runner adapter (#4518): the seam a differential test plugs a decode path into. The spine judges an ENGINE runner's trace against a REFERENCE runner's trace, so both a golden implementation and the path under test satisfy the same interface. Real adapters — a llama.cpp reference, a fak engine mode — wire in here behind Run without changing the harness.

type SamplingParams 

type SamplingParams struct {
	Temperature float64 `json:"temperature"`
	TopK        int     `json:"top_k,omitempty"`
	TopP        float64 `json:"top_p,omitempty"`
	MaxTokens   int     `json:"max_tokens"`
	Seed        int64   `json:"seed,omitempty"`
}

SamplingParams is the decode configuration a case is pinned to. It is part of the replay contract: a result is only reproducible if the params that produced it travel with it. Seed is honored by stochastic runners; a temperature-zero case must decode identically regardless of seed (frozen by #4525).

type SamplingRunner 

type SamplingRunner struct {
	Label  string
	Tokens []string
	Probs  []float64
}

SamplingRunner is the engine-side adapter for distribution cases: it draws Params.MaxTokens samples from ITS OWN categorical distribution (Tokens/Probs) using the case's fixed seed. A faithful engine carries the same distribution as the reference; a biased engine carries a skewed one — the deterministic mutant source the tests use to prove the gate trips.

func (SamplingRunner) Name 

func (s SamplingRunner) Name() string

func (SamplingRunner) Run 

func (s SamplingRunner) Run(c QualityCase) (Trace, error)

type SchedulerParity 

type SchedulerParity struct{}

SchedulerParity is the differential oracle for scheduler-policy output parity (#4537): every request's output under the engine's policy must equal its output under the reference policy token for token — policy may reorder EXECUTION, never a request's own tokens. Outputs are compared request by request in the reference body's canonical (submission) order, so the first corrupted token localizes to a request, a step within it, and a flat index into the canonical stream, and the Detail names the policy that corrupted it.

func (SchedulerParity) Judge 

func (SchedulerParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (SchedulerParity) Kind 

func (SchedulerParity) Kind() string

func (SchedulerParity) Name 

func (SchedulerParity) Name() string

type SchedulerRunner 

type SchedulerRunner struct {
	Label  string
	Policy string
	// contains filtered or unexported fields
}

SchedulerRunner executes the case's workload (parsed from the Prompt) under one scheduling policy. The zero defect is a faithful scheduler: it may run requests in any policy order because each request decodes into its own fresh buffer. The defect field (set via SchedulerEngine) injects the shared-buffer bug this child exists to catch.

func SchedulerEngine 

func SchedulerEngine(policy, defect string) SchedulerRunner

SchedulerEngine returns a scheduler runner for policy with an optional injected defect: "" schedules faithfully (isolated per-request buffers); "shared-buffer" reuses one un-cleared slab across the batch so a reordering policy corrupts the shorter requests' outputs with a longer request's stale tail. This is the deterministic mutant source the tests use to prove the parity gate trips.

func (SchedulerRunner) Name 

func (s SchedulerRunner) Name() string

func (SchedulerRunner) Run 

func (s SchedulerRunner) Run(c QualityCase) (Trace, error)

type ScriptedRunner 

type ScriptedRunner struct {
	Label string
	Trace Trace
}

ScriptedRunner is an engine-path adapter that replays a fixed trace. It models a specific engine build/mode for hermetic tests and for the CLI demo, and is the shape real engine adapters take: capture the engine's decode into a Trace and return it. The spine never trusts the runner's self-description — only its emitted tokens/text, judged against the reference.

func DemoEngine 

func DemoEngine(defect string) ScriptedRunner

DemoEngine returns an engine runner for the demo case with an optional injected defect: "" reproduces the reference (clean pass); "decode" flips one token so the greedy differential oracle fails at that index; "stop" decodes past the reference's last token so the failure localizes to the stop decision; "report" corrupts the text so the grounding rubric fails on a forbidden/omitted claim. This is the deterministic mutant source the spine test and CLI use to prove each gate trips.

func DtypeDeltaEngine 

func DtypeDeltaEngine(defect string) ScriptedRunner

DtypeDeltaEngine returns an engine runner for the dtype fixture with an optional injected defect: "" executes every dtype lane faithfully (honest per-dtype rounding only); "fp16-blowout" pushes one fp16 logit 4 bands past the reference at step dtFP16DefectStep — the half-precision kernel drift this child exists to catch; "bf16-band" drifts one bf16 logit by a delta the fp16 band would absorb, proving the budgets are per-dtype. These are the deterministic mutant sources the tests use to prove the gate trips.

func (ScriptedRunner) Name 

func (s ScriptedRunner) Name() string

func (ScriptedRunner) Run 

func (s ScriptedRunner) Run(_ QualityCase) (Trace, error)

type SearchArm added in v0.44.0 

type SearchArm struct {
	ID       string   `json:"id"`
	Point    string   `json:"point"`
	P        float64  `json:"p_value"`
	Evidence Evidence `json:"evidence"`
}

SearchArm is one candidate point the sweep evaluated: the threshold, variant, or comparison it names, the p-value that point produced, and the spine Evidence it came from. Every arm the search touched belongs here, including the ones that lost — the losers are the family, and the family is what the winner is priced against.

type SeedReplay 

type SeedReplay struct{}

SeedReplay is the differential oracle for seeded replay (#4529): the engine's sampled token stream must equal the reference stream token by token, because both were pinned to the same seed. Any mismatch — a drifted seed, a step-dependent bug, a truncated decode — is reported as the FIRST divergence, so "the sampler is nondeterministic" localizes to "step 3 drew 'flux' where the reference drew 'ember'".

func (SeedReplay) Judge 

func (SeedReplay) Judge(ref, eng Trace, c QualityCase) Verdict

func (SeedReplay) Kind 

func (SeedReplay) Kind() string

func (SeedReplay) Name 

func (SeedReplay) Name() string

type SeededRunner 

type SeededRunner struct {
	Label string
	// contains filtered or unexported fields
}

SeededRunner decodes by running the deterministic sampler for the case's Params.MaxTokens steps under Params.Seed. The zero value is a faithful engine; the defect field (set via SeedReplayEngine) injects a nondeterminism bug. It is the ScriptedRunner-style adapter for the stochastic replay seam: a real sampled engine wires in behind the same Runner interface and is judged the same way.

func SeedReplayEngine 

func SeedReplayEngine(defect string) SeededRunner

SeedReplayEngine returns a seeded engine runner with an optional injected nondeterminism defect: "" decodes faithfully under the case's pinned seed; "seed-drift" ignores the seed (samples under seed+1) so the decode diverges from the first differing draw; "step-bug" corrupts the single token at step 3 so the first divergence localizes mid-sequence. This is the deterministic mutant source the tests use to prove the replay gate trips.

func (SeededRunner) Name 

func (s SeededRunner) Name() string

func (SeededRunner) Run 

func (s SeededRunner) Run(c QualityCase) (Trace, error)

type SelectionAdjustment added in v0.44.0 

type SelectionAdjustment string

SelectionAdjustment is the closed set of policies that may price the search. AdjustNone is admitted only so the cost of skipping the price can be MEASURED (see the null simulation in the test file) — a sweep that declares it is refused. 

const (
	// AdjustSidak is the Sidak correction on the minimum p-value, 1 - (1 - p)^m.
	// Exact for INDEPENDENT arms; anti-conservative under positive dependence.
	AdjustSidak SelectionAdjustment = "sidak"
	// AdjustBonferroni is min(1, m*p): valid under arbitrary dependence between
	// arms, and therefore the honest default for a sweep over one shared sample.
	AdjustBonferroni SelectionAdjustment = "bonferroni"
	// AdjustNone prices nothing. It bounds nothing.
	AdjustNone SelectionAdjustment = "none"
)

type SelectionRule added in v0.44.0 

type SelectionRule string

SelectionRule is the closed set of rules by which a sweep may pick the arm it retains. The rule is what makes the price computable: pricing the minimum of m p-values is only correct if the minimum is what was actually kept. 

const (
	// SelectMinP retains the most significant arm. This is a search, and it is
	// priced over the whole family.
	SelectMinP SelectionRule = "min-p"
	// SelectPredeclared names the absence of a search: exactly one point, fixed
	// before any of the evidence was seen. It is priced at m = 1 — that is, not at
	// all — which is precisely the claim a sweep is not entitled to make.
	SelectPredeclared SelectionRule = "predeclared"
)

type SpecDecodeParity 

type SpecDecodeParity struct{}

SpecDecodeParity is the speculative-decoding differential oracle (#4539): the engine's speculative token stream must equal the target-only reference stream exactly, because a correct accept/reject rule is exactness-preserving. Any mismatch — a wrongly-kept draft token, a dropped fallback, a truncated block — is reported as the FIRST divergence, so "speculative decode changed the output" localizes to "token 2 kept 'fern' where target-only decode emits 'elm'".

func (SpecDecodeParity) Judge 

func (SpecDecodeParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (SpecDecodeParity) Kind 

func (SpecDecodeParity) Kind() string

func (SpecDecodeParity) Name 

func (SpecDecodeParity) Name() string

type SpecDecodeRunner 

type SpecDecodeRunner struct {
	Label string
	// contains filtered or unexported fields
}

SpecDecodeRunner is the speculative engine adapter: it decodes the case's Params.MaxTokens steps under Params.Seed through the speculative loop. The zero value is a faithful engine; the defect field (set via SpecDecodeEngine) injects the lenient accept rule. Real speculative engine adapters wire in behind the same Runner interface and are judged the same way.

func SpecDecodeEngine 

func SpecDecodeEngine(defect string) SpecDecodeRunner

SpecDecodeEngine returns a speculative engine runner with an optional injected defect: "" verifies faithfully (rejections fall back to the target token, so the stream equals target-only decode); "lenient-accept" keeps draft tokens the target would reject, so the stream first departs at specFirstDriftStep. This is the deterministic mutant source the tests use to prove the parity gate trips.

func (SpecDecodeRunner) Name 

func (s SpecDecodeRunner) Name() string

func (SpecDecodeRunner) Run 

func (s SpecDecodeRunner) Run(c QualityCase) (Trace, error)

type StopTruncation 

type StopTruncation struct{}

StopTruncation is the stop-semantics differential oracle (#4528): it verifies the engine honored the case's termination contract against the reference — the truncation cap (Params.MaxTokens), the hard stop token / EOS sentinel, the case's stop strings (Rubric.Forbidden entries, which should have halted generation before they reached the assembled text), and, when no earlier stop fired, termination at the same step as the reference. A decode that keeps talking past its stop is a stop-semantics defect even when every emitted token individually matches greedy truth, so this gate is separate from (and complementary to) greedy-token-diff.

func (StopTruncation) Judge 

func (StopTruncation) Judge(ref, eng Trace, c QualityCase) Verdict

func (StopTruncation) Kind 

func (StopTruncation) Kind() string

func (StopTruncation) Name 

func (StopTruncation) Name() string

type Suite added in v0.42.0 

type Suite struct {
	Tier            Tier        `json:"tier"`
	Cases           []SuiteCase `json:"cases"`
	TotalRuntimeSec int64       `json:"total_runtime_seconds"`
	TotalTimeoutSec int64       `json:"total_timeout_seconds"`
	MaxMemoryMiB    int64       `json:"max_memory_mib"`
	MaxCPU          int         `json:"max_cpu"`
	MaxAccelerators int         `json:"max_accelerators"`
}

Suite is one tier's ordered case list plus its summed cost. Cases are ordered cheapest-evidence-first so the suite fails fast: the quickest check that can localize a defect runs before the expensive ones.

type SuiteCase added in v0.42.0 

type SuiteCase struct {
	CaseID string         `json:"case_id"`
	Family EvidenceFamily `json:"family"`
	Owner  string         `json:"owner"`
	Tier   Tier           `json:"tier"`
	Cost   CostSpec       `json:"cost"`
}

SuiteCase is one placed case's routing header: enough for an operator to read who owns it, what it proves, which tier runs it, and what it costs — without opening the full case.

type SuitePlan added in v0.42.0 

type SuitePlan struct {
	Schema   string        `json:"schema"`
	Suites   []Suite       `json:"suites"`
	Rejected []SuiteReject `json:"rejected,omitempty"`
}

SuitePlan is the machine-readable output of a split: the three ordered suites and every rejected case. It is a pure function of (cases, budgets) — same corpus, same plan — so a split replays.

func SplitCorpus added in v0.42.0 

func SplitCorpus(fsys fs.FS, dir string, budgets map[Tier]TierBudget) (SuitePlan, error)

SplitCorpus loads every *.json case under dir in fsys and splits it into the PR / nightly / release suites (see SplitSuites; nil budgets = DefaultBudgets).

A file that fails the canonical case contract becomes a SuiteReject naming the file rather than an error: one broken case must not stop the rest of the corpus from being routed, and it must not disappear either. An error is returned only when the corpus DIRECTORY itself cannot be read — an infrastructure failure, which is never reported as a quality verdict.

func SplitSuites added in v0.42.0 

func SplitSuites(cases []QualityCase, budgets map[Tier]TierBudget) SuitePlan

SplitSuites partitions a corpus into PR / nightly / release suites under the given budgets (nil = DefaultBudgets). A case is rejected — never placed — when it fails the canonical routing contract (bad/absent tier, timeout, resources, owner, or family) or when its declared cost exceeds its tier's budget. Placed cases are ordered cheapest-evidence-first within each suite; the suites themselves are returned in fixed PR, nightly, release order so the plan is deterministic.

func (SuitePlan) Green added in v0.42.0 

func (p SuitePlan) Green() (bool, string)

Green is the fail-closed verdict on a split plan, for a caller that gates on it (a CI step, an operator readout) rather than reading the whole plan. It is true only when the split rejected nothing AND actually placed evidence: a plan that refused a case is not a pass — refusing it is the point — and a plan whose suites are all empty is not a pass either, because a corpus that qualifies nothing has produced no evidence. The returned string names the first blocking reason, in the plan's deterministic case order, so a caller can print it and act on it.

type SuiteReject added in v0.42.0 

type SuiteReject struct {
	CaseID string `json:"case_id"`
	Tier   Tier   `json:"tier,omitempty"`
	Reason string `json:"reason"`
}

SuiteReject is one case the split refused, with the reason it could not be routed. A rejected case is never placed in a suite — missing routing evidence or evidence too expensive for its tier is never a pass.

type SweepCertification added in v0.44.0 

type SweepCertification struct {
	Alpha        float64             `json:"alpha"`
	Rule         SelectionRule       `json:"rule"`
	Adjustment   SelectionAdjustment `json:"adjustment"`
	DeclaredArms int                 `json:"declared_arms"`
	Winner       string              `json:"winner"`
}

SweepCertification is the declaration a sweep is held to: the error budget, the rule that picked the winner, the adjustment that prices the search, the size of the COMPLETE family that was searched, and which arm was retained. DeclaredArms is stated separately from the arms handed over on purpose — it is the operator's assertion about how wide the search was, and cross-checking it against the arms on the record is what catches an understated family.

type TZDeterminism 

type TZDeterminism struct{}

TZDeterminism is the temperature-zero determinism oracle (#4525): every request surface in the engine trace must equal the greedy reference token by token. Any mismatch — injected sampling noise, a surface-specific decode path, a truncated stream — is reported as the FIRST divergence with the offending surface named, so "streaming sometimes answers differently at temp 0" localizes to "surface streaming, token 2".

func (TZDeterminism) Judge 

func (TZDeterminism) Judge(ref, eng Trace, c QualityCase) Verdict

func (TZDeterminism) Kind 

func (TZDeterminism) Kind() string

func (TZDeterminism) Name 

func (TZDeterminism) Name() string

type TZSurfaceRunner 

type TZSurfaceRunner struct {
	Label string
	// contains filtered or unexported fields
}

TZSurfaceRunner is the multi-surface engine adapter: it decodes the case once per request surface and returns all streams in one enveloped Trace. The zero value is a faithful engine; the defect field (set via TZEngine) models a temp-0 path that still injects sampling noise on one surface.

func TZEngine 

func TZEngine(defect string) TZSurfaceRunner

TZEngine returns a multi-surface engine with an optional injected defect: "" decodes every surface through the shared greedy path (clean pass); "sampling-noise" injects a sampled token on the streaming surface at step tzNoiseStep so exactly one surface departs from the greedy reference. This is the deterministic mutant source the tests use to prove the gate trips.

func (TZSurfaceRunner) Name 

func (r TZSurfaceRunner) Name() string

func (TZSurfaceRunner) Run 

func (r TZSurfaceRunner) Run(c QualityCase) (Trace, error)

type ThresholdAuditRequest added in v0.44.0 

type ThresholdAuditRequest struct {
	Threshold              float64   `json:"threshold"`
	Comparison             string    `json:"comparison"`
	Observations           []float64 `json:"observations"`
	BoundaryWidth          *float64  `json:"boundary_width"`
	RoundTripDecimalPlaces *int      `json:"round_trip_decimal_places"`
	Perturbation           *float64  `json:"perturbation"`
}

ThresholdAuditRequest preserves the evidence needed to decide whether a threshold verdict survives supported numeric round-trips and plausible input perturbations. Pointer fields distinguish an explicit zero from absent evidence.

type ThresholdAuditResult added in v0.44.0 

type ThresholdAuditResult struct {
	Schema                string  `json:"schema"`
	Verdict               string  `json:"verdict"`
	RefusalCode           string  `json:"refusal_code,omitempty"`
	Reason                string  `json:"reason"`
	ObservationCount      int     `json:"observation_count"`
	BoundaryCount         int     `json:"boundary_count"`
	BoundaryMass          float64 `json:"boundary_mass"`
	RoundTripFlipCount    int     `json:"round_trip_flip_count"`
	PerturbationFlipCount int     `json:"perturbation_flip_count"`
}

ThresholdAuditResult is the stable, machine-readable receipt for a threshold audit. A conclusion is accepted only when every supplied observation keeps its membership under both checks.

func AuditThreshold added in v0.44.0 

func AuditThreshold(req ThresholdAuditRequest) ThresholdAuditResult

AuditThreshold refuses conclusions when required evidence is absent or when supported precision/perturbation checks can change threshold membership.

type Tier 

type Tier string

Tier is the cost/cadence class a required quality case is assigned to. The release gate aggregates evidence per tier so a cheap PR check is never confused with a nightly hardware suite — the PR / nightly / release separation top stacks keep (#4578 scope). 

const (
	TierPR      Tier = "pr"
	TierNightly Tier = "nightly"
	TierRelease Tier = "release"
)

type TierBudget added in v0.42.0 

type TierBudget struct {
	Tier            Tier
	MaxTimeout      int64 // seconds; 0 = unbounded
	MaxAccelerators int   // peak accelerators the tier admits
}

TierBudget is the evidence-cost ceiling one tier admits. A tier is a cost class: the PR tier buys a fast, CPU-only signal on every push; nightly buys a longer CPU budget for sampling and corpora; release buys unbounded time and the accelerators GPU-parity and hardware qualification need. A case whose declared cost exceeds its tier's budget is refused — that refusal IS the "split by evidence cost" contract.

type TierCost added in v0.44.0 

type TierCost struct {
	Tier        Tier    `json:"tier"`
	Comparisons int     `json:"comparisons"`
	CostSeconds float64 `json:"cost_seconds"`
}

TierCost is one tier's share of the family's documented evidence cost, so an operator can read what the grid costs per cadence rather than as one number (#4568 acceptance: assign a tier and document runtime/resource cost).

type TierSpec 

type TierSpec struct {
	Name string `json:"name"`
}

TierSpec routes a case to exactly one validation cadence.

type TokParityRunner 

type TokParityRunner struct {
	Label string
	// contains filtered or unexported fields
}

TokParityRunner renders the case's message list through the shared pure template. The zero value is a faithful engine; the defect field (set via TokenizerParityEngine) injects one template-drift bug into the rendered ids. It is the ScriptedRunner-style adapter for the tokenization seam: a real engine tokenizer wires in behind the same Runner interface and is judged the same way.

func TokenizerParityEngine 

func TokenizerParityEngine(defect string) TokParityRunner

TokenizerParityEngine returns a template engine runner with an optional injected defect: "" renders faithfully (id-exact with the pinned reference); each tokParityDefect* constant injects that one drift class. This is the deterministic mutant source the tests use to prove the parity gate trips per defect class.

func (TokParityRunner) Name 

func (r TokParityRunner) Name() string

func (TokParityRunner) Run 

func (r TokParityRunner) Run(c QualityCase) (Trace, error)

type TokenizerParity 

type TokenizerParity struct{}

TokenizerParity is the differential oracle for #4521: the engine's rendered token-id sequence must equal the pinned reference sequence exactly — special tokens and content ids alike. Any mismatch is reported as the FIRST divergent id, annotated with its special-token marker when it has one, so "the chat template drifted" localizes to "id 0 was <|system|> where the reference put <bos>".

func (TokenizerParity) Judge 

func (TokenizerParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (TokenizerParity) Kind 

func (TokenizerParity) Kind() string

func (TokenizerParity) Name 

func (TokenizerParity) Name() string

type ToleranceSpec 

type ToleranceSpec struct {
	Metric   string  `json:"metric"`
	Absolute float64 `json:"absolute,omitempty"`
	Relative float64 `json:"relative,omitempty"`
	Revision string  `json:"revision"`
}

ToleranceSpec names the tolerance policy and its immutable source.

type TopKTopPBoundary 

type TopKTopPBoundary struct{}

topk_topp.go — #4526: exercise top-k and top-p boundary semantics.

The candidate-set truncation math is where fluent-but-wrong sampling starts: an off-by-one at the k-th slot, a nondeterministic tie-break, or an exclusive p-boundary comparison all change WHICH tokens are even eligible, yet every downstream draw still looks plausible. This oracle validates the truncation itself: the case's Reference carries the full vocabulary (Reference.Tokens) and one aligned logits row (Reference.Logits[0]); the engine emits the candidate set it kept as its Trace.Tokens; and Judge compares that against the correctly computed boundary set for the case's Params.TopK / Params.TopP.

The truncation contract, stated precisely (every rule below is deterministic and closed over the case inputs):

  • Probabilities are the softmax of the logits row. Tokens are ranked by DESCENDING probability; TIES are resolved by ASCENDING vocabulary index (the token appearing earlier in Reference.Tokens wins the higher rank).
  • top-k keeps EXACTLY the k highest-ranked tokens. TopK <= 0 disables the filter (the zero value of SamplingParams.TopK means unset); TopK >= vocab keeps the entire vocabulary.
  • top-p keeps the SMALLEST prefix of the (post-top-k, renormalized) ranking whose cumulative probability reaches p; a token landing EXACTLY on the p boundary is included and truncation happens immediately after it (the comparison is >=, with a tiny epsilon so float rounding cannot flip an exact boundary into an exclusion). TopP <= 0 disables the filter (the zero value of SamplingParams.TopP means unset); TopP >= 1 keeps the whole surviving set, since only the full set carries the target mass.
  • The kept set is an ORDERED trace — the survivors in ranking order — so the comparison is a plain token diff and a defect localizes to the first slot that disagrees, exactly like the greedy differential.

func (TopKTopPBoundary) Judge 

func (TopKTopPBoundary) Judge(ref, eng Trace, c QualityCase) Verdict

func (TopKTopPBoundary) Kind 

func (TopKTopPBoundary) Kind() string

func (TopKTopPBoundary) Name 

func (TopKTopPBoundary) Name() string

type TpParity 

type TpParity struct{}

TpParity is the differential oracle for #4542: the sharded engine's token stream must equal the TP=1 reference stream exactly, and where both traces carry per-step logits those must agree within tpLogitTolerance. A correct partitioned reduction (stable shard order) reproduces TP=1 tokens exactly and logits to within reassociation ulps; a reduction-order/associativity bug that materially changes a logit flips a token, and the verdict pins the FIRST step it happened at — "TP=8 broke the report" localizes to "token 2 decoded 'fern' where TP=1 decoded 'cinder'".

func (TpParity) Judge 

func (TpParity) Judge(ref, eng Trace, _ QualityCase) Verdict

func (TpParity) Kind 

func (TpParity) Kind() string

func (TpParity) Name 

func (TpParity) Name() string

type TpShardedRunner 

type TpShardedRunner struct {
	Label  string
	Degree int
	// contains filtered or unexported fields
}

TpShardedRunner decodes under tensor-parallel degree Degree via the partitioned reduction. The zero defect is a faithful engine; the defect field (set via TpEngine) injects the reduction-order bug. It is the ScriptedRunner-style adapter for the TP seam: a real sharded engine wires in behind the same Runner interface and is judged the same way.

func TpEngine 

func TpEngine(degree int, defect string) TpShardedRunner

TpEngine returns a tensor-parallel engine runner at the given degree with an optional injected defect: "" reduces in stable order (token-exact parity with TP=1, logits within fp tolerance at ANY degree); "reduce-reorder" reverses shard 0's fp summation order at step tpBugStep so the decoded token there flips from the winner to the runner-up. This is the deterministic mutant source the tests use to prove the parity gate trips.

func (TpShardedRunner) Name 

func (r TpShardedRunner) Name() string

func (TpShardedRunner) Run 

func (r TpShardedRunner) Run(c QualityCase) (Trace, error)

type Trace 

type Trace struct {
	Runner string      `json:"runner"`
	Tokens []string    `json:"tokens"`
	Logits [][]float64 `json:"logits,omitempty"`
	Text   string      `json:"text"`
}

Trace is the captured output of one decode path: the emitted token sequence, optional per-step logits (top candidates), and the assembled text. Tokens are the primary differential surface — text scoring happens only after the token stream is proven equal or explicitly allowed to differ, because two paths can assemble identical-looking text from divergent tokens.

type TrackedIssue added in v0.44.0 

type TrackedIssue struct {
	Key          RegressionKey `json:"key"`
	Marker       string        `json:"marker"`
	Kind         string        `json:"kind"`
	Open         bool          `json:"open"`
	Occurrences  int           `json:"occurrences"`
	GreenRuns    int           `json:"green_runs"`
	FirstSeenRun string        `json:"first_seen_run"`
	LastFailRun  string        `json:"last_fail_run"`
	LastRun      string        `json:"last_run"`
}

TrackedIssue is the tracker's durable per-key record: the issue's identity, whether it is open, how many runs have observed it failing, and how many consecutive green runs of its coordinates it has accumulated toward closure.

type Tracker added in v0.44.0 

type Tracker struct {
	Schema      string                  `json:"schema"`
	GreenWindow int                     `json:"green_window"`
	Issues      map[string]TrackedIssue `json:"issues,omitempty"`
}

Tracker is the durable, JSON-serializable ledger the lifecycle is computed against — the memory that makes "repeated runs update ONE issue" possible. A driver persists it beside the suite and reloads it on the next run.

func NewTracker added in v0.44.0 

func NewTracker(greenWindow int) *Tracker

NewTracker returns an empty tracker closing on the given green window. A non-positive window is clamped to DefaultGreenWindow: closing on zero green runs would close every issue the moment it stopped being observed, which is the exact "silence is a pass" failure this file exists to refuse.

func (*Tracker) Apply added in v0.44.0 

func (t *Tracker) Apply(plan FilingPlan, file Filer) FilingReport

Apply executes a plan through a Filer and advances the tracker for exactly the filings that took effect. A Hold has no external effect and always advances; an Open, Update, or Close advances only once the Filer reports it landed. A nil Filer lands nothing external — a dry run that still advances holds, so an operator can rehearse a plan without inventing filings that do not exist.

func (*Tracker) OpenIssues added in v0.44.0 

func (t *Tracker) OpenIssues() []TrackedIssue

OpenIssues returns the tracker's currently-open issues in marker order.

func (*Tracker) Plan added in v0.44.0 

func (t *Tracker) Plan(run FilingRun, obs []Observation) FilingPlan

Plan folds one run's observations into the lifecycle actions they imply, WITHOUT mutating the tracker. Findings sharing a key are deduplicated into one filing (a retried case in the same run is the same finding, not two), a keyed finding opens or updates its one issue, and an open issue whose coordinates went green this run holds — or closes, once the green window is complete.

An open issue whose coordinates were not observed at all is deliberately untouched: an unrun case is not a green case, so it neither advances toward closure nor counts as a new occurrence.

type Verdict 

type Verdict struct {
	Oracle          string      `json:"oracle"`
	Kind            string      `json:"kind"`
	Pass            bool        `json:"pass"`
	Score           float64     `json:"score,omitempty"`
	FirstDivergence *Divergence `json:"first_divergence,omitempty"`
	Detail          string      `json:"detail"`
}

Verdict is one oracle's decision. FirstDivergence is set by differential oracles to the exact step where the engine first departed from the reference — the localizing evidence the epic exists to produce. Score is set by rubric oracles.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.