quality

Documentation

Overview

Package quality is the subjective quality + value-for-money assessment layer for iterion's live e2e tests. It runs a cross-family LLM judge panel over the REAL work product a bot/feature run produced (a git diff, the board issues it created, the docs it edited, the findings it surfaced…) together with the run's price metrics (cost, tokens, duration), and emits a structured Snapshot. Snapshots are persisted as an append-only per-target history (see store.go) so successive runs can be compared — primarily against the last — to attest improvement or regression of a bot's quality/value over time.

Goodhart resistance is the whole point of the design:

• The assessed bot never sees this rubric or these judges; the assessment is external + post-hoc, so the bot cannot optimise to it and it is NEVER wired back into any bot loop.
• Judges grade the real artifact (the diff/work), never the bot's self-reported claims.
• The panel is cross-family (two model families) so no single-model bias dominates; a judge sharing the bot's family is flagged.
• Scoring is multi-dimensional + narrative, not one gameable number, and the headline comparison is RELATIVE to the previous snapshot (more reliable than absolute scoring across non-deterministic runs).

The deterministic half (snapshot store, compare, regression gate) lives in store.go and is unit-tested without any LLM call. The live half (the judge panel) is exercised only from the //go:build live e2e tests.

Index

• Constants
• Variables
• func DefaultJudgeModels() []string
• type Aggregate
  • func RunPanelWith(ctx context.Context, models []string, invoke JudgeInvoker, ev Evidence, ...) (Aggregate, error)
• type Delta
  • func Compare(prev, cur *Snapshot) *Delta
  • func (d *Delta) IsRegression(tol float64) (bool, []string)
• type Dimension
• type Evidence
• type JudgeInvoker
  • func ClawInvoker(reg *model.Registry) JudgeInvoker
• type JudgeVerdict
• type Metrics
• type Relative
• type Snapshot
  • func (s *Snapshot) Overall() float64
  • func (s *Snapshot) Value() float64
• type SnapshotStore
  • func NewSnapshotStore(dir string) *SnapshotStore
  • func (s *SnapshotStore) Dir(name string) string
  • func (s *SnapshotStore) Last(name string) (*Snapshot, bool, error)
  • func (s *SnapshotStore) List(name string) ([]string, error)
  • func (s *SnapshotStore) Load(path string) (*Snapshot, error)
  • func (s *SnapshotStore) Write(snap *Snapshot) (string, error)

Constants

const (
	// DefaultSameBand is the |Δ| within which an overall change is called
	// "same" rather than better/worse — absorbs LLM-judge noise.
	DefaultSameBand = 0.05
	// DefaultRegressTolerance is how far overall (or value) must drop below
	// the previous snapshot before the opt-in gate fails the test. Wider
	// than the same-band so only clear regressions trip it.
	DefaultRegressTolerance = 0.08
)

const SchemaVersion = 1

SchemaVersion is bumped when the Snapshot JSON shape changes incompatibly, so a reader can detect and skip stale history files.

Variables

var Dimensions = []Dimension{
	DimEfficacy, DimCompleteness, DimOutputQuality, DimRestraint,
	DimReliability, DimValueForMoney, DimOverall,
}

Dimensions is the canonical ordered rubric.

Functions

func DefaultJudgeModels

func DefaultJudgeModels() []string

DefaultJudgeModels returns the cross-family judge panel. Override with ITERION_LIVE_JUDGE_MODELS (comma-separated provider/model specs). The default deliberately spans two families (Anthropic + OpenAI) so the panel is cross-family out of the box.

Types

type Aggregate

type Aggregate struct {
	Verdicts     []JudgeVerdict        `json:"verdicts"`
	MeanScores   map[Dimension]float64 `json:"mean_scores"`
	Disagreement map[Dimension]float64 `json:"disagreement,omitempty"` // max-min spread per dimension
	Note         string                `json:"note,omitempty"`         // e.g. judges skipped, family overlap
}

Aggregate is the combined panel result.

func RunPanelWith

func RunPanelWith(ctx context.Context, models []string, invoke JudgeInvoker, ev Evidence, prev *Snapshot) (Aggregate, error)

RunPanelWith runs every judge model over the evidence via invoke and aggregates their verdicts. A judge that errors is skipped with a note rather than failing the panel, so a single available family still yields an assessment. An error is returned only when NO judge produced a verdict.

type Delta

type Delta struct {
	PrevRunID    string                `json:"prev_run_id"`
	OverallDelta float64               `json:"overall_delta"` // cur.overall - prev.overall
	ValueDelta   float64               `json:"value_delta"`   // cur.value_for_money - prev
	PerDimension map[Dimension]float64 `json:"per_dimension"` // cur - prev, per dimension
	Verdict      Relative              `json:"verdict"`       // headline (overall) better/same/worse
}

Delta is the deterministic comparison of a current snapshot against the previous one. It is descriptive; the regression decision (with a tolerance) is IsRegression so the same Delta can drive both reporting and an opt-in gate.

func Compare

func Compare(prev, cur *Snapshot) *Delta

Compare returns nil when prev is nil (no baseline yet); otherwise a per-dimension delta with a headline verdict on the overall score.

func (*Delta) IsRegression

func (d *Delta) IsRegression(tol float64) (bool, []string)

IsRegression reports whether the overall OR value-for-money score dropped more than tol below the previous snapshot, with human-readable reasons. A nil Delta (no baseline) is never a regression.

type Dimension

type Dimension string

Dimension is one axis of the stable quality rubric. The set is fixed so snapshots stay comparable across runs; a multi-dimensional rubric (vs a single score) is itself a Goodhart safeguard.

const (
	DimEfficacy      Dimension = "efficacy"        // did it truly accomplish the task (grounded in the artifact)
	DimCompleteness  Dimension = "completeness"    // full scope, no façade/stub/partial
	DimOutputQuality Dimension = "output_quality"  // idiomatic, maintainable, correct, no over-engineering
	DimRestraint     Dimension = "restraint"       // minimal diff, no unrequested churn or scope creep
	DimReliability   Dimension = "reliability"     // clean convergence, no thrash, acceptable termination
	DimValueForMoney Dimension = "value_for_money" // quality achieved per $ + time spent
	DimOverall       Dimension = "overall"         // holistic, NOT a mechanical average
)

type Evidence

type Evidence struct {
	Kind          string  // "bot" | "feature"
	Name          string  // target name, e.g. "review-pr" or "permission"
	Persona       string  // optional persona, e.g. "Revi"
	PrimaryFamily string  // the assessed run's primary model family ("anthropic", "openai", …)
	Task          string  // scenario / vars summary — what the run was asked to do
	WorkProduct   string  // the REAL artifact: git diff, board issues JSON, doc diff, findings…
	Outcome       string  // run status + acceptable-error reason + which nodes finished
	Metrics       Metrics // price side
}

Evidence is the real-artifact bundle handed to every judge. The caller (the e2e glue) gathers it from the run — the engine never inspects a workspace or a store itself, keeping it pure and provider-agnostic.

type JudgeInvoker

type JudgeInvoker func(ctx context.Context, modelSpec, system, userMsg string, schema json.RawMessage) (map[string]any, error)

JudgeInvoker runs ONE judge: given a model spec, the rubric system prompt, the rendered evidence user message, and the forced-tool JSON schema, it returns the judge's structured verdict object. Abstracting the call lets callers route different judges through different backends — e.g. an OpenAI judge via claw's direct-generation path and an Anthropic judge via the claude_code OAuth delegate (so a true cross-family panel works without an ANTHROPIC_API_KEY). See ClawInvoker for the default.

func ClawInvoker

func ClawInvoker(reg *model.Registry) JudgeInvoker

ClawInvoker is the default judge invoker: it resolves the model spec to a claw client and forces the structured-output tool. Requires an API key for the model's provider (claw cannot use Claude Code OAuth).

type JudgeVerdict

type JudgeVerdict struct {
	Model             string                 `json:"model"`
	Family            string                 `json:"family"`
	SameFamilyAsBot   bool                   `json:"same_family_as_bot"`
	Scores            map[Dimension]float64  `json:"scores"`
	Narrative         string                 `json:"narrative"`
	RelativeVsPrev    map[Dimension]Relative `json:"relative_vs_prev,omitempty"`
	RelativeNarrative string                 `json:"relative_narrative,omitempty"`
	Confidence        float64                `json:"confidence"`
}

JudgeVerdict is one judge's structured assessment.

type Metrics

type Metrics struct {
	CostUSD    float64 `json:"cost_usd"`
	Tokens     int     `json:"tokens"`
	DurationMS int64   `json:"duration_ms"`
	Iterations int     `json:"iterations"`
	ModelCalls int     `json:"model_calls"`
	Retries    int     `json:"retries"`
}

Metrics is the price side of value-for-money: what the run consumed.

type Relative

type Relative string

Relative is a per-dimension judgment of the current run versus the previous snapshot.

const (
	RelBetter Relative = "better"
	RelSame   Relative = "same"
	RelWorse  Relative = "worse"
	RelNA     Relative = "n/a" // no comparable previous snapshot
)

type Snapshot

type Snapshot struct {
	SchemaVersion  int       `json:"schema_version"`
	Kind           string    `json:"kind"` // "bot" | "feature"
	Name           string    `json:"name"`
	Persona        string    `json:"persona,omitempty"`
	RunID          string    `json:"run_id"`
	At             time.Time `json:"at"`
	BotVersion     string    `json:"bot_version,omitempty"`
	IterionSHA     string    `json:"iterion_sha,omitempty"`
	Task           string    `json:"task,omitempty"`
	Metrics        Metrics   `json:"metrics"`
	Aggregate      Aggregate `json:"aggregate"`
	EvidenceDigest string    `json:"evidence_digest,omitempty"` // diffstat / counts / report excerpt
	PrevRunID      string    `json:"prev_run_id,omitempty"`
	Comparison     *Delta    `json:"comparison,omitempty"` // vs the previous snapshot, computed deterministically
}

Snapshot is one assessed run, persisted as a single JSON file in the target's append-only history directory. It bundles the price metrics, the panel aggregate, an evidence digest, and (when a prior snapshot existed) a deterministic comparison against it.

func (*Snapshot) Overall

func (s *Snapshot) Overall() float64

Overall returns the aggregated overall score (0 if absent).

func (*Snapshot) Value

func (s *Snapshot) Value() float64

Value returns the aggregated value-for-money score (0 if absent).

type SnapshotStore

type SnapshotStore struct {
	Root string
}

SnapshotStore is the committed, append-only history root. In the e2e suite Root is e2e/testdata/live/quality; each target gets a subdir of per-run JSON files named "<UTC-ts>__<runid>.json" so lexical order is chronological.

func NewSnapshotStore

func NewSnapshotStore(dir string) *SnapshotStore

NewSnapshotStore roots a store at dir.

func (*SnapshotStore) Dir

func (s *SnapshotStore) Dir(name string) string

Dir returns the (sanitised) per-target history directory.

func (*SnapshotStore) Last

func (s *SnapshotStore) Last(name string) (*Snapshot, bool, error)

Last returns the most recent prior snapshot for a target (the history tail), or (nil,false,nil) when none exists.

func (*SnapshotStore) List

func (s *SnapshotStore) List(name string) ([]string, error)

List returns the target's history file paths in chronological (lexical) order. A missing directory yields an empty slice, not an error.

func (*SnapshotStore) Load

func (s *SnapshotStore) Load(path string) (*Snapshot, error)

Load reads a single snapshot file.

func (*SnapshotStore) Write

func (s *SnapshotStore) Write(snap *Snapshot) (string, error)

Write persists snap as a new history file and returns its path. It is deterministic given the snapshot (filename derives from At+RunID), so re-writing the same snapshot is idempotent.