trust

package
v1.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package trust is the confidence layer of the Trust Control Plane: it measures how much each evidence source's verdict is actually worth (calibration) and how costly a wrong answer is for a given task (defect cost). Phase 1 ships the calibration engine and the defect-cost model; the SPRT optimal-stopping ensemble that consumes them lands in Phase 2.

Index

Constants

View Source 
const FixedSwarmBaseline = 5

FixedSwarmBaseline is the fixed fan-out an SPRT run is compared against.

Variables

This section is empty.

Functions

func ClusterByAgreement added in v1.3.1 

func ClusterByAgreement(texts []string, equiv AnswerEquivalence) [][]int

ClusterByAgreement greedily groups indices into texts whose entries equiv treats as the same answer — shared by Run's own co-agreement recording and any ensembling caller (e.g. swarm.CalibratedJudge) that needs the same grouping.

func DefaultCoAgreementPath added in v1.3.1 

func DefaultCoAgreementPath() string

DefaultCoAgreementPath is where co-agreement observations are persisted.

func DefaultLogPath 

func DefaultLogPath() string

DefaultLogPath is where SPRT runs are persisted (~/.hydra/trust.jsonl).

func DefaultPath 

func DefaultPath() string

DefaultPath is where calibration is persisted for the CLI (~/.hydra/calibration.jsonl).

func FalseConsensusWarning added in v1.3.1 

func FalseConsensusWarning(path, family string) (j float64, warn bool)

FalseConsensusWarning reports whether family's measured coupling has crossed criticalCoupling — its members are effectively one vote.

func FamilyCoupling added in v1.3.1 

func FamilyCoupling(path, family string) (j float64, ok bool)

FamilyCoupling reports family's measured excess same-family agreement: the empirical rate at which two DIFFERENT same-family sources agree, minus the rate at which two different-family sources agree — the measurable definition of "these are correlated, not independently confirming." ok is false below minCoAgreementSamples same-family pairs.

func FamilyDiscount added in v1.3.1 

func FamilyDiscount(path, family string) float64

FamilyDiscount replaces the flat correlation constant: 1-J, so a family with no measured excess correlation (J=0) is not discounted at all, and a family whose members are nearly always identical (J→1) contributes almost nothing on a repeat vote. Falls back to defaultCorrelationDiscount below minCoAgreementSamples.

func KnownFamilies added in v1.3.1 

func KnownFamilies(path string) []string

KnownFamilies returns the distinct families observed in the co-agreement log, sorted — for a caller (e.g. `hyctl trust calibration`) that wants to check every family's FalseConsensusWarning without knowing names up front.

func LogRun 

func LogRun(path string, r RunLog) error

LogRun appends one run to the trust log, stamping TS and Config (best-effort) if the caller left them blank.

func RecordCoAgreement added in v1.3.1 

func RecordCoAgreement(path, domain string, ids, families, texts []string, equiv AnswerEquivalence)

RecordCoAgreement clusters one task's answers by agreement and appends the observation — best-effort, since a logging failure must never affect the ensemble it observes. Sources with an empty Family are dropped: a repeat vote is only ever discounted when a family is known, so an unfamilied source carries no correlation signal to record.

func TaskHash 

func TaskHash(prompt string) string

TaskHash is a short stable identifier for a prompt, used to correlate a run with `hyctl trust explain <task_hash>`.

func TextEquivalence added in v1.1.0 

func TextEquivalence(candidate, answer string) bool

TextEquivalence is the v1 default agreement check: case-insensitive with collapsed whitespace. It removes trivial-formatting false disagreements but is NOT semantic — two behaviorally-equivalent answers with different identifiers still register as disagreement. Supply WithEquivalence for behavior-based comparison.

Types

type Answer 

type Answer struct {
	Text    string
	CostUSD float64 // actual cost if known; falls back to Source.EstCostUSD
}

Answer is what a source returned for the task.

type AnswerEquivalence added in v1.1.0 

type AnswerEquivalence func(candidate, answer string) bool

AnswerEquivalence decides whether a source's answer counts as agreeing with the current candidate for the purpose of accumulating correctness evidence.

The v1 default (TextEquivalence) compares normalized text — which miscounts two independently-correct answers that differ only in wording (variable names, println vs. fmt.Println) as *disagreement*, pushing Λ the wrong way. In the real benchmark this capped achieved confidence at 32.9% even though both sources were oracle-verified correct (see TRUST_CONTROL_PLANE_BENCHMARK_FINDINGS.md §3). Callers that can compare *behavior* — an oracle verdict in a benchmark, an LLM judge in production — inject a semantic comparator via WithEquivalence.

type BenchCase 

type BenchCase struct {
	Label       string  `json:"label"`
	Accuracy    float64 `json:"accuracy"`     // fraction of runs that returned the correct answer
	MeanSamples float64 `json:"mean_samples"` // average model calls
	SavedPct    float64 `json:"saved_pct"`    // vs a fixed-N swarm
}

BenchCase is the measured result for one task difficulty.

type BenchmarkResult 

type BenchmarkResult struct {
	Trials  int         `json:"trials"`
	FixedN  int         `json:"fixed_n"`
	Cases   []BenchCase `json:"cases"`
	Blended BenchCase   `json:"blended"` // 71% easy / 29% hard task mix
}

BenchmarkResult is the output of Benchmark.

func Benchmark 

func Benchmark(trials int, seed int64) BenchmarkResult

Benchmark runs the real SPRT ensemble (Run) over synthetic sources of known reliability and measures samples-vs-fixed-N and accuracy. It is the [MEASURED] counterpart to the Manifesto's [MODEL] Law 3 numbers — deterministic for a given seed. Easy tasks use 90%-reliable sources; hard tasks 74%.

type Calibrator 

type Calibrator struct {
	// contains filtered or unexported fields
}

Calibrator maintains an online confusion posterior per (source, domain) and derives calibrated LLR and diagnostic power from it. Safe for concurrent use.

func New 

func New(path string) (*Calibrator, error)

New constructs a Calibrator. When path is non-empty it replays any existing records from that file so calibration survives process restarts; subsequent updates are appended there. An empty path keeps everything in memory.

func (*Calibrator) D 

func (c *Calibrator) D(source, domain string) float64

D is the diagnostic power of a source (nats): the expected LLR of its verdict given a truly-correct item, i.e. KL(Bern(se) ‖ Bern(1−sp)). It is ≥0 always, and 0 exactly when se+sp=1 (the source carries no information — Law 2). Use it to order which source to sample next (most evidence first).

func (*Calibrator) LLR 

func (c *Calibrator) LLR(source, domain string, saidCorrect bool) float64

LLR returns the calibrated log-likelihood-ratio contribution (nats) of a verdict from this source. Positive nats are evidence the answer is correct.

  • says correct: ln( se / (1−sp) )
  • says incorrect: ln( (1−se) / sp )

An uncalibrated / coin-flip source (se+sp≈1) yields LLR≈0.

func (*Calibrator) Report 

func (c *Calibrator) Report() []Stat

Report returns per-(source,domain) calibration stats, most-diagnostic first.

func (*Calibrator) Update 

func (c *Calibrator) Update(source, domain string, saidCorrect bool, actual Outcome) error

Update records one observation: a source said correct/incorrect and the ground-truth outcome came back. It updates the posterior and, when a path is set, appends the event so it survives restarts. OutcomeUnknown is ignored.

type Decision 

type Decision int

Decision is how an SPRT run ended. 

const (
	// DecisionAccept: Λ crossed the accept threshold A at the target confidence.
	DecisionAccept Decision = iota
	// DecisionStoppedOnBudget: ran out of budget/heads before reaching A — the
	// residual uncertainty is where a human oracle (review) should be spent.
	DecisionStoppedOnBudget
)

func (Decision) String 

func (d Decision) String() string

type DefectModel 

type DefectModel struct {
	W DefectWeights
	// ToleratedLeakUSD is the expected leaked-defect cost held constant by
	// RequiredConfidence. Zero uses defaultToleratedLeakUSD.
	ToleratedLeakUSD float64
}

DefectModel prices the cost of shipping an incorrect answer for a task. That cost sets how much confidence a task must clear before Hydra stops sampling (RequiredConfidence) and is surfaced in `hyctl dispatch --confidence --file`, `hyctl trust defect`, and `hyctl dispatch --dry-run`.

func NewDefectModel 

func NewDefectModel() *DefectModel

NewDefectModel returns a model using the default weights.

func (*DefectModel) CostUSD 

func (d *DefectModel) CostUSD(t Task) float64

CostUSD = Base × blast × w_irrev × w_pii × w_prod, with each risk factor applied only when present. BlastRadius ≤ 0 is treated as 1.0 (local).

func (*DefectModel) RequiredConfidence 

func (d *DefectModel) RequiredConfidence(t Task) float64

RequiredConfidence maps a task's defect cost to the confidence-of-correctness Hydra should demand before it stops sampling. It holds the *expected leaked defect cost* constant: α = toleratedLeak / defectCost, so target = 1 − α. A costlier mistake linearly lowers the tolerated error probability; the result is clamped to [0.5, maxConfidence].

type DefectWeights 

type DefectWeights struct {
	Base         float64 // baseline cost of any wrong answer, USD
	Irreversible float64 // multiplier when the change can't be cheaply undone
	PII          float64 // multiplier when personal data is involved
	Production   float64 // multiplier when the target is production
}

DefectWeights are the multipliers that turn a task's risk attributes into the dollar cost of shipping a wrong answer. They are deliberately simple and tunable; the values below are illustrative defaults, not measured constants. Graphify will supply real blast-radius multipliers in Phase 3.

func DefaultWeights 

func DefaultWeights() DefectWeights

DefaultWeights returns the built-in defect weights. A local, reversible, non-PII, non-prod mistake costs Base; each risk factor scales it up.

type Evidence 

type Evidence struct {
	Source      string  `json:"source"`
	Agreed      bool    `json:"agreed"` // matched the candidate at the time
	LLR         float64 `json:"llr"`    // calibrated contribution (nats), after any discount
	Candidate   string  `json:"candidate"`
	LambdaAfter float64 `json:"lambda_after"`
	CostUSD     float64 `json:"cost_usd"`

	// ConfidenceAfter is σ(LambdaAfter) — the running P(correct) once this
	// source had been weighed. Stored rather than left for readers to derive:
	// the ledger is a public JSON type written to trust.jsonl and read by the
	// run log, and a second copy of the sigmoid in each reader is a third place
	// for the confidence to drift.
	ConfidenceAfter float64 `json:"confidence_after"`
}

Evidence is one entry in the LLR ledger — a single source's calibrated contribution to the running log-odds that the candidate is correct.

type Executor 

type Executor interface {
	Execute(ctx context.Context, src Source, task Task) (Answer, error)
}

Executor runs one source against a task. Production wraps the swarm executor; tests inject a deterministic fake.

type Outcome 

type Outcome int

Outcome is the ground-truth-ish label used to train calibration. 

const (
	// OutcomeUnknown means no ground truth is available yet — it never trains.
	OutcomeUnknown Outcome = iota
	// OutcomeCorrect: tests passed / user approved / not reverted within N days.
	OutcomeCorrect
	// OutcomeIncorrect: tests failed / user rejected / reverted.
	OutcomeIncorrect
)

func ParseOutcome 

func ParseOutcome(s string) Outcome

ParseOutcome maps a CLI string to an Outcome. Unrecognized → OutcomeUnknown.

type Result 

type Result struct {
	Candidate  string     `json:"candidate"`
	Confidence float64    `json:"confidence"` // σ(Λ)
	Decision   Decision   `json:"decision"`
	Lambda     float64    `json:"lambda"`
	SpentUSD   float64    `json:"spent_usd"`
	Samples    int        `json:"samples"`
	Ledger     []Evidence `json:"ledger"`
}

Result is the outcome of Run.

func Run 

func Run(ctx context.Context, task Task, sources []Source, exec Executor, cal *Calibrator, t Target, opts ...RunOption) (*Result, error)

Run executes the sequential probability ratio test: it samples sources in decreasing evidence-per-dollar order, accumulating the calibrated log-likelihood ratio Λ that the current candidate answer is correct, and stops as soon as Λ crosses the Wald accept threshold A (target confidence) or runs out of budget. A source that disagrees with the candidate pushes Λ down; if Λ crosses the reject threshold B the run pivots to that source's answer (destructive interference collapses to the better branch).

type RunLog 

type RunLog struct {
	TS         string     `json:"ts"`
	TaskHash   string     `json:"task_hash"`
	Domain     string     `json:"domain"`
	TargetConf float64    `json:"target_conf"`
	FinalConf  float64    `json:"final_conf"`
	Samples    int        `json:"samples"`
	Models     []string   `json:"models"`
	CostUSD    float64    `json:"cost_usd"`
	CostSource string     `json:"cost_source"` // from cost.SourceLabels
	Decision   string     `json:"decision"`    // accept | stopped_on_budget
	Ledger     []Evidence `json:"ledger,omitempty"`
	Config     string     `json:"config,omitempty"` // deployment-identity breadcrumb (config.Breadcrumb)
}

RunLog is one persisted SPRT run — the data the "By The Numbers" page graduates from [MODEL] to [MEASURED], and what `hyctl trust stats/explain` read.

func LoadRuns 

func LoadRuns(path string) ([]RunLog, error)

LoadRuns reads all runs from the trust log. A missing file yields no runs.

type RunOption added in v1.1.0 

type RunOption func(*runConfig)

RunOption configures an SPRT run without changing Run's core signature.

func WithEquivalence added in v1.1.0 

func WithEquivalence(fn AnswerEquivalence) RunOption

WithEquivalence overrides how answer agreement is decided (default: TextEquivalence). A nil comparator is ignored, keeping the default.

type Source 

type Source struct {
	ID         string // calibration key, e.g. "model:claude-sonnet"
	Family     string // base-model family, for the correlation discount ("" = independent)
	EstCostUSD float64
}

Source is the minimal view of a head that SPRT needs. Kept independent of provider.Head so the trust package stays a low-level dependency; callers (swarm/dispatch) adapt their heads to this in Phase 2b.

type Stat 

type Stat struct {
	Source string  `json:"source"`
	Domain string  `json:"domain"`
	N      float64 `json:"n"`  // real observations (excludes prior)
	Se     float64 `json:"se"` // sensitivity
	Sp     float64 `json:"sp"` // specificity
	D      float64 `json:"d"`  // diagnostic power (nats), expected |LLR|
}

Stat is one row of a calibration report — the human/JSON-facing view.

type Stats 

type Stats struct {
	Runs            int     `json:"runs"`
	MeanSamples     float64 `json:"mean_samples"`
	FixedSwarmN     int     `json:"fixed_swarm_n"` // baseline for comparison
	SamplesSavedPct float64 `json:"samples_saved_pct"`
	AutoClearedPct  float64 `json:"auto_cleared_pct"` // reached accept without a human
	MeanTargetConf  float64 `json:"mean_target_conf"`
	MeanFinalConf   float64 `json:"mean_final_conf"`
	TotalCostUSD    float64 `json:"total_cost_usd"`
}

Stats is the aggregate view produced by `hyctl trust stats`.

func Aggregate 

func Aggregate(runs []RunLog, fixedN int) Stats

Aggregate summarizes a set of runs against a fixed-N swarm baseline.

type Target 

type Target struct {
	Confidence float64 // desired P(correct), e.g. 0.95 → α = 1-conf
	MaxCostUSD float64 // hard spend ceiling; 0 = no limit
}

Target configures an SPRT run. The domain used for calibration lookups comes from the Task, not here, so there is exactly one source of truth for it.

type Task 

type Task struct {
	Domain string
	// BlastRadius scales cost by how much other code a wrong answer would break.
	// 0 or 1 = local/self-contained. Graphify supplies real values in Phase 3;
	// until then callers leave it at the default (treated as 1.0).
	BlastRadius float64
	// Irreversible: the change cannot be cheaply undone (data migration, deploy).
	Irreversible bool
	// TouchesPII: the task handles personal data — a wrong answer is a leak risk.
	TouchesPII bool
	// Production: the change targets a production surface, not a scratch/dev one.
	Production bool
}

Task describes the work whose wrong-answer cost the DefectModel prices.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.