Documentation
¶
Overview ¶
Package verifier builds and scores the claim graph (§11).
Three stages, in order: retrieve candidate pairs worth comparing, infer the edge between each pair, then derive confidence from the resulting graph. The split is what keeps the expensive stage small — every candidate pair costs a model call, so retrieval decides the bill.
§11.1's rule shapes all of it: the Verifier operates over the session's whole claim set, not the batch one lead produced. A contradiction between two claims found by different leads is the case rev 1 structurally could not see, and it is the case that matters most.
Index ¶
- Constants
- Variables
- func Batches(pairs []Pair, size int) [][]Pair
- func CandidatePairs(ctx context.Context, r Retriever, targets, pool []*core.Claim, maxPerClaim int, ...) (needJudging []Pair, decided []Judged, err error)
- func Edges(sessionID string, verdicts []Judged, stalenessGap time.Duration) []core.ClaimEdge
- func JudgePairs(ctx context.Context, p llm.Provider, model string, pairs []Pair, batchSize int, ...) ([]Judged, []Pair)
- func PublisherOf(source string) string
- type Cluster
- type Effect
- type FollowUp
- type FollowUpOptions
- type GroundOutcome
- type GroundReport
- type GroundResult
- type Grounder
- type Judged
- type LexicalRetriever
- type Pair
- type Relation
- type Result
- type Retriever
- type Score
- type SourceClass
- type Verifier
Constants ¶
const DefaultBatchSize = 8
DefaultBatchSize is how many pairs go into one adjudication call.
A tradeoff rather than an optimum. Larger batches amortize the instructions further, but a small model's attention degrades across many parallel judgements and its output ceiling arrives sooner — the same 3B model that truncated mid-JSON during claim mining is the one that will be asked to emit eight verdicts. Partial responses are salvaged per verdict for exactly that reason, so an over-long batch degrades rather than failing.
DefaultGroundShareOfEscrow is the fraction of released escrow grounding may spend.
A guess, flagged as one, in the same family as the fan-out taper and the verification share. What it protects is not a guess: the report must still be affordable afterwards, and §8.3's whole argument for escrow is that research otherwise leaves nothing to write the answer with. Grounding is research.
const DefaultMaxCandidates = 8
DefaultMaxCandidates bounds how many pairs one claim can put up for adjudication.
The cap, not a similarity threshold, is what bounds cost. A threshold would need a number nobody can justify, and picking it wrong silently drops real edges; the cap costs at most this many calls per claim and is honest about being arbitrary.
const DefaultMaxFollowUpsPerPass = 2
DefaultMaxFollowUpsPerPass bounds how many leads one verification pass may propose.
Small on purpose. A pass runs on the replan cadence, so follow-ups accumulate across passes anyway, and a graph full of disagreement could otherwise queue more verification work in one go than the session has budget to run.
const DefaultMaxFollowUpsPerRoot = 3
DefaultMaxFollowUpsPerRoot is §11.4's per-root cap, so one stubborn claim cannot consume the session.
Counted across the whole session rather than per pass. A disagreement that survives three attempts to settle it is a real disagreement, and the honest thing is to report it as disputed rather than to keep paying for the same answer.
const DefaultMaxGroundChecks = 5
DefaultMaxGroundChecks bounds how many claims one pass may re-read.
Small, and the count is the point rather than the value. Grounding is the most expensive check in the system — a fetch, an extraction and a model call per claim — and it runs against the released escrow, which is the money the report needs. A pass that grounds everything produces a well-checked set of claims and no report to put them in.
DefaultMaxShareOfBudget is the cumulative verification allowance.
0.25 is a guess and flagged as one, in the same category as §9.1's fan-out taper: nothing here can yet say what fraction of a session SHOULD go to verification, because that depends on how often the graph changes a conclusion — which is what §14.2's corpus measures. Set generously on purpose; a ceiling that binds before the graph is useful would make the Verifier look worthless for reasons of arithmetic.
const DefaultMaxVerifyDepth = 3
DefaultMaxVerifyDepth is §11.4's chain cap: a follow-up of a follow-up of a follow-up is drift, not diligence.
const DefaultStalenessGap = 365 * 24 * time.Hour
DefaultStalenessGap is how far apart two publication dates must be before a contradiction reads as staleness rather than disagreement (§11.2).
A guess, and flagged as one. Two papers a week apart that disagree are disagreeing; two a decade apart usually are not, and the boundary between them depends on how fast the field moves — §11.3 calls this "the question's volatility", and nothing here can measure it yet. §14.2's corpus is what would calibrate it.
const MaxClaimChars = 600
MaxClaimChars bounds one claim's contribution to the prompt.
A page that produced a 40kB "claim" would otherwise set the size of every batch it appears in. The extractor caps claim length already; this is the boundary that does not trust it.
const MaxPassageChars = 2400
MaxPassageChars bounds the window that reaches the model.
A page is attacker-controlled in the sense that matters — mole followed a search result to reach it — so its length must not decide the size of a call. contextAround already windows it; this is the boundary that does not trust that.
Variables ¶
var ErrNoProvider = errors.New("verifier: no model provider configured")
ErrNoProvider means the Verifier was built without an LLM.
Functions ¶
func Batches ¶
Batches splits pairs into model calls.
Batching is the difference between verification costing a third of the session and costing a twentieth. Measured on a real 13-claim run: 96 ordered pairs dedupe to 54, and one call per pair spends roughly 39k tokens against a run that spent 114k on the research itself. Batched at 8 it is seven calls and about 6k.
The instruction block is what dominates a single-pair call — around 350 tokens of rules against 60 of claim text — so amortizing it over several pairs is where nearly all of the saving comes from.
func CandidatePairs ¶
func CandidatePairs( ctx context.Context, r Retriever, targets, pool []*core.Claim, maxPerClaim int, judged func(pairKey string) bool, ) (needJudging []Pair, decided []Judged, err error)
CandidatePairs builds the work list for one verification pass.
Returns the pairs that need a model call, plus the ones settled without one. targets are the claims not yet verified; pool is the session's whole claim set, because §11.1's rule is that the Verifier sees the store rather than the batch one lead produced.
judged reports whether a pair already carries an edge; nil means none do. Pairs are generated once per claim in the normal course — a pair (older, newer) is produced when newer is verified — so this only matters when a pass is re-run over a session that was interrupted.
func Edges ¶
Edges turns verdicts into graph edges.
Two things happen here rather than at the model boundary. `unrelated` produces nothing — it is a real answer, and storing it would fill the edge table with rows recording absence. And a contradiction between claims published far enough apart becomes `supersedes` (§11.2): a 2019 finding contradicted by a 2025 one is usually stale rather than disputed.
The supersedes edge REPLACES the contradiction rather than joining it. Keeping both would have §11.3 penalize the newer claim's confidence for a disagreement it wins, and the direction of the supersedes edge already records the conflict.
func JudgePairs ¶
func JudgePairs(ctx context.Context, p llm.Provider, model string, pairs []Pair, batchSize int, log *slog.Logger) ([]Judged, []Pair)
JudgePairs adjudicates pairs with no ledger and no store.
The evaluation path. Verifier.Run reserves budget, settles it, writes edges and marks claims verified — all correct for research and all wrong for asking "would a different model have judged these better". Charging an eval re-run to the session would inflate its spend and corrupt cost-per-claim for the very session being examined, and writing edges would destroy the judgements being compared against.
So this shares the prompt, the batching and the parsing with the real path, and shares nothing else. The cost is the evaluator's, not the session's, and it is not recorded — which is a deliberate exception to §8.1's every-call-writes-a-row rule, taken because the alternative is worse.
Returns what it judged and what it could not, in that order.
func PublisherOf ¶
PublisherOf reduces a source to the entity that published it.
eTLD+1 via the public suffix list, which is the whole point: §11.3 counts "distinct domains/publishers, not distinct URLs", and the naive last-two-labels shortcut gets that wrong in both directions. It merges bbc.co.uk and theguardian.co.uk into one publisher named "co.uk", and it splits alice.github.io from bob.github.io — which the PSL correctly treats as separate publishers, since github.io is a suffix.
fetch.DomainOf deliberately keeps the naive version: it is a grouping key for the §10.4 failure ranking, where merging a country's sites is untidy rather than wrong, and changing it would alter the meaning of already-stored rows. Here a wrong answer inflates confidence, so it is worth the list.
Types ¶
type Cluster ¶
type Cluster struct {
// Claims, ordered by ID so a cluster's identity does not depend on traversal
// order.
Claims []*core.Claim
}
Cluster is a set of claims that duplicate_of edges say are one assertion.
func Clusters ¶
Clusters groups claims by duplicate_of edges.
Union-find over the duplicate edges only. `supports` deliberately does not merge: two claims that support each other are two pieces of evidence, and merging them would delete exactly the corroboration §11.3 counts.
func (Cluster) Representative ¶
Representative is the claim that stands for the cluster in a report.
§11.2: a duplicate_of cluster renders as one claim with N sources. The pick is the strongest source class, then the earliest claim, so it is deterministic and prefers the best-attested wording.
type Effect ¶
type Effect string
Effect is what a relation actually changes about the graph.
The live taxonomy is now one relation per effect, so for anything the model says today this is a rename. It earns its place on the boundary: retired relations still arrive from claim_edges rows and from stored pair-set dumps, and comparing a five-relation graph against a three-relation one is exactly the measurement that justified shrinking the taxonomy in the first place.
The asymmetry it was written to expose, kept because it is the reason the vocabulary is the size it is. EdgeContradicts is read by confidence derivation (a penalty), by the executor (it queues a follow-up lead), by grounding, by the report's duplicate collapse and by the scorecard. EdgeDuplicateOf drives clustering, and therefore the publisher count corroboration is computed from. EdgeSupports and EdgeRefines were read in exactly two places: the validity switch in core, and the display order in `mole trace`.
const ( // EffectContradiction penalizes confidence and queues a follow-up lead. EffectContradiction Effect = "contradiction" // EffectDuplicate merges claims into a cluster, changing the publisher count. EffectDuplicate Effect = "duplicate" // EffectInert changes nothing derived. Displayed, never consumed. EffectInert Effect = "inert" )
type FollowUp ¶
type FollowUp struct {
Lead core.Lead
// Because names the contradiction that prompted it, for the trace.
Because string
}
FollowUp is a lead the Verifier wants run, and why.
func FollowUps ¶
func FollowUps(verdicts []Judged, opts FollowUpOptions) []FollowUp
FollowUps proposes leads to settle the contradictions in a set of verdicts.
Deterministic: verdicts are considered in canonical pair order and the output is sorted, so a cassette replays and two runs of one session queue the same work.
type FollowUpOptions ¶
type FollowUpOptions struct {
SessionID string
ActorType core.ActorType
// MaxDepth is the chain cap. Zero takes DefaultMaxVerifyDepth.
MaxDepth int
// MaxPerRoot bounds follow-ups per root claim. Zero takes the default.
MaxPerRoot int
// StalenessGap must match the one Edges uses, or a pair the graph resolved as
// staleness is still researched as a live disagreement. Zero takes the default.
StalenessGap time.Duration
// MaxTotal bounds one pass's output, so a graph full of disagreement cannot
// queue more work than the session can run. Zero means unbounded here — the
// budget ceiling still applies downstream.
MaxTotal int
// ExistingPerRoot is how many follow-ups each root already has, so the
// per-root cap counts the session rather than the pass.
ExistingPerRoot map[string]int
}
FollowUpOptions bounds lead generation.
type GroundOutcome ¶
type GroundOutcome string
GroundOutcome is what one check learned.
const ( // GroundConfirmed: the quote is in the source and supports the claim. GroundConfirmed GroundOutcome = "confirmed" // GroundUnsupported: the quote is there, and does not support the claim. The // serious verdict — this is quote-mining or a misreading, and §11.3 penalizes // it near-fatally. GroundUnsupported GroundOutcome = "unsupported" // GroundVanished: the quote is no longer in the source. // // NOT evidence against the claim. The quote was checked verbatim against the // fetched text at extraction time, so its absence now means the page changed, // and scoring that as "the evidence does not support the claim" would penalize // a claim for a publisher's edit. Leaves Grounded nil. GroundVanished GroundOutcome = "vanished" // GroundUnreachable: the source could not be re-read. Nothing was learned, so // nothing is written about the claim beyond the note. GroundUnreachable GroundOutcome = "unreachable" // GroundUndecided: the judge could not answer. Same treatment as unreachable — // a check that failed is not a claim that failed. GroundUndecided GroundOutcome = "undecided" )
type GroundReport ¶
type GroundReport struct {
Checked int
Confirmed int
Unsupported int
Vanished int
Unreachable int
Undecided int
// Fetches is how many sources were actually re-read. Lower than Checked when
// several claims cite one source: the document is fetched once and every claim
// on it is checked against the same text.
Fetches int
Calls int
Spent int64
Results []GroundResult
// Skipped is how many candidates the caps left unchecked.
Skipped int
// NotFetchable is how many claims cite a page the search provider supplied, which
// grounding cannot re-read without manufacturing a mismatch.
NotFetchable int
// Degraded says what the pass could not finish (§9.5).
Degraded string
}
GroundReport is what a grounding pass did.
type GroundResult ¶
type GroundResult struct {
ClaimID string
Outcome GroundOutcome
// Note is one line for the trace. Model prose about page text, flattened, never
// the page text itself.
Note string
}
GroundResult is one claim's check.
type Grounder ¶
Grounder re-reads sources. Split out so a pass can run without a network at all, and so the fetcher the actor already configured — with its robots handling, its rate limiter and its SSRF guard — is the one used here.
type Judged ¶
type Judged struct {
Pair
Relation Relation
Weight float64
Rationale string
// DecidedBy names what produced the verdict — a model, or the mechanical rule
// that made a call without one. Written to claim_edges.created_by, so a trace
// can say why an edge exists.
DecidedBy string
}
Judged is a pair with a verdict.
type LexicalRetriever ¶
type LexicalRetriever struct{}
LexicalRetriever scores claim pairs by IDF-weighted cosine similarity over their words.
IDF is computed over the session's own claims, which is what makes this work without tuning. Every claim in a session about MambaByte contains "MambaByte", so that term carries almost no information about which claims relate to each other; "subword", "tokenization" and "1.31" carry most of it. A retriever weighting all shared words equally ranks by topic and returns the whole session.
Cosine rather than raw overlap because claim length varies: a long claim shares more words with everything by accident, and would otherwise dominate every candidate list.
type Pair ¶
Pair is two claims put up for adjudication, in canonical order.
A.ID < B.ID always. Not cosmetic: a pair reached from both ends is one pair, and the whole point of canonicalizing is that it gets judged and stored once.
type Relation ¶
type Relation string
Relation is what an adjudicator may say about a pair of claims.
Deliberately not core.EdgeKind. Two differences, both load-bearing:
- `unrelated` is a valid and common verdict that stores no edge. Making it representable is what lets a pass distinguish "judged, no relation" from "not judged", without an edge table full of rows recording absence.
- `supersedes` is absent. §11.2 derives it from PublishedAt, and a model cannot see publication dates — asking it to rank two claims by recency invites invention.
const ( RelContradicts Relation = "contradicts" RelDuplicate Relation = "duplicate_of" // RelNeither is the catch-all, and the most common answer by a wide margin. // // Named "neither" rather than "unrelated" on purpose. Two claims can be closely // related — one evidence for the other, one a sharper version of the other — and // still belong here, because duplication and contradiction are the only // distinctions anything downstream acts on. Asking a model to file an obviously // supporting pair under "unrelated" fights its own sense of the words, and the // measurement says it loses that fight: it reached for "supports" on six pairs // that were nothing of the kind. RelNeither Relation = "neither" )
The relations a model is asked for. Three, because three is what the graph reads.
Retired relations, still accepted on the way in.
These were offered to the model until the labelled pair set showed what they cost. Over 37 pairs, claude-haiku-4-5 scored 76% raw and 97% by effect: eight of its nine errors were choosing between "supports", "refines" and "unrelated", and every one of those built an edge nothing reads. Five of six self-inconsistencies were the same shuffle. The distinction bought nothing and was the dominant source of error.
Kept parseable for two reasons that outlive the change: claim_edges rows written before this carry the old kinds, and a model that answers "supports" out of habit should have its verdict normalized rather than thrown away — the effect is identical, and discarding it would lose a real judgement over vocabulary.
func (Relation) EffectOf ¶
EffectOf maps a relation to what it changes.
Kept beside Edges deliberately: Edges is the function that gives relations their consequences, so if a "supports" edge ever starts feeding a derivation, the two have to be changed together or this becomes a lie.
type Result ¶
type Result struct {
ClaimsVerified int
PairsRetrieved int
// PairsDecidedFree were settled by a mechanical rule, with no model call.
PairsDecidedFree int
PairsJudged int
// PairsConfirmed and PairsUnconfirmed count the second judgement's outcome when
// ConfirmEdges is on. Unconfirmed pairs kept their claims and lost their edge.
PairsConfirmed int
PairsUnconfirmed int
// PairsUnjudged is pairs a batch did not return a verdict for, plus pairs
// skipped because the allowance ran out.
PairsUnjudged int
EdgesWritten int
Calls int
Spent int64
// ClaimsScored is how many claims had confidence recomputed. Larger than
// ClaimsVerified whenever a new claim changed an older claim's standing.
ClaimsScored int
// Scores is the per-cluster derivation, for a trace. §11.3 requires the number
// be explainable, and a bare 0.62 is exactly the figure people either trust
// blindly or dismiss.
Scores []Score
// Contradictions is how many live disagreements this pass found. The number the
// planner most needs: a sub-question whose evidence is disputed is not answered.
Contradictions int
// FollowUps are leads the Verifier wants run to settle them (§11.4). The caller
// queues them — the Verifier does not own the lead queue, and one that did
// could not be replayed against a cassette.
FollowUps []FollowUp
// Degraded says what the pass could not finish (§9.5). Empty when complete.
Degraded string
// contains filtered or unexported fields
}
Result reports what one pass did.
type Retriever ¶
type Retriever interface {
// Candidates returns up to max claims from pool that may relate to target,
// most promising first. Never returns target itself.
Candidates(ctx context.Context, target *core.Claim, pool []*core.Claim, max int) ([]*core.Claim, error)
}
Retriever finds the claims worth comparing against a target.
A cheap filter in front of an expensive judgement. Recall is what matters here and precision barely does: a spurious candidate costs one adjudication and comes back labelled `unrelated`, while a missed one is an edge the graph never has and a contradiction nobody sees.
An interface because §11.2 leaves the embedding question open, and the sketch files it as a cost-versus-dependency decision (per-claim embedding cost against the weight of another provider). LexicalRetriever answers it with "not yet": free, deterministic, replayable, and it needs no network. If eval data shows embeddings retrieve pairs this misses, they drop in here.
type Score ¶
type Score struct {
Confidence float64
// Publishers is the number of DISTINCT publishers supporting the assertion —
// registrable domains, not URLs. Five pages on one site is one publisher, and
// counting them as five is the corroboration inflation the sketch names as an
// open risk.
Publishers int
// Class is the strongest source class in the cluster.
Class SourceClass
// Contradictors is the number of distinct publishers whose claims contradict
// this one. Distinct publishers again, for the same reason.
Contradictors int
// Superseded is set when a later-published claim supersedes this one (§11.2).
Superseded bool
// Grounded mirrors the cluster's grounding result: nil where §11.5's budgeted
// re-fetch never ran, which is most claims.
Grounded *bool
// Explain is the derivation, one line, for a trace.
Explain string
// Cluster is the claims this score covers, representative resolvable via
// Cluster.Representative.
//
// Carried so a caller does not re-derive what scoreCluster already computed.
// output.Findings independently recomputed distinct publishers, superseded-as-target
// and cross-cluster contradictions from the same inputs; the two agreed only because
// both were written carefully, and nothing enforced it.
Cluster Cluster
}
Score is one cluster's derived confidence and the terms that produced it.
The breakdown is not decoration. §11.3 requires the number be explainable in the trace viewer, and a bare 0.62 is exactly the kind of figure people either trust blindly or dismiss. Every field here is something a reader can check.
func DeriveConfidence ¶
DeriveConfidence scores every claim in a session from the graph.
Every member of a duplicate cluster gets the cluster's score, because they are the same assertion: giving them different numbers would make the report's choice of wording change its stated confidence.
type SourceClass ¶
type SourceClass string
SourceClass is how much weight a publisher's output carries on its own.
const ( ClassPeerReviewed SourceClass = "peer-reviewed" ClassPrimary SourceClass = "primary" ClassSecondary SourceClass = "secondary" // ClassUnknown is the default, and defaulting here rather than to something // optimistic is the point: an unlisted domain is treated as the weakest kind of // evidence, so failing to recognize a good source understates confidence rather // than inventing it. ClassUnknown SourceClass = "unknown" )
func ClassOf ¶
func ClassOf(source string) SourceClass
ClassOf classifies a source.
A hand-written list, and it cannot classify the web — that is not the claim. The claim is that it is better than treating every source alike, that it is deterministic and testable, and that its default is the WEAK class, so an unrecognized publisher understates confidence rather than inventing it.
Matched against the registrable domain, never a substring of the URL. That distinction is load-bearing: "https://nih.gov.attacker.example/paper" contains ".gov" and must not be classified as a government source, and a substring match is exactly how it would be.
func (SourceClass) Weight ¶
func (c SourceClass) Weight() float64
Weight is the multiplier a class contributes.
type Verifier ¶
type Verifier struct {
Store store.Store
Ledger *budget.Ledger
LLM llm.Provider
// Model overrides the tier's default for every call this Verifier makes. Empty
// leaves the cheap-tier model in place.
//
// Separate from the tier because the workloads are not alike. Chunk mining is
// extraction and runs per chunk; adjudication decides whether two sentences can both
// be true and runs per batch — 31 calls against 7 on a live 25-claim run. A model too
// expensive to mine with can be affordable to judge with, and judging is the stage
// whose errors corrupt confidence, the report's disagreements and three eval metrics.
Model string
// Retriever selects candidate pairs. Nil takes LexicalRetriever.
Retriever Retriever
// MaxCandidatesPerClaim bounds fan-out per claim. Zero takes the default.
MaxCandidatesPerClaim int
// BatchSize is how many pairs go into one adjudication call. Zero takes the
// default.
BatchSize int
// ConfirmEdges re-judges every verdict that would build an edge and keeps only
// the ones a second call agrees with. See confirm() for the measurement behind
// it: 51% precision becomes 70%, at 53 edges where one judgement kept 105.
ConfirmEdges bool
// StalenessGap is how far apart two publication dates must be for a
// contradiction to read as staleness (§11.2). Zero takes the default.
StalenessGap time.Duration
// session's whole budget. Zero takes DefaultMaxShareOfBudget.
//
// A ceiling is not optional here. Pair count grows with the square of the
// claim count before the per-claim cap bites, so a session that gathers a lot
// of claims can spend more verifying them than it spent finding them. On the
// real 13-claim run, one call per pair would have cost about a third of the
// whole session.
MaxShareOfBudget float64
// MaxVerifyDepth and MaxFollowUpsPerRoot are §11.4's caps. Zero takes the
// defaults.
MaxVerifyDepth int
MaxFollowUpsPerRoot int
// MaxFollowUpsPerPass bounds how much work one pass may queue. Zero takes
// DefaultMaxFollowUpsPerPass.
MaxFollowUpsPerPass int
// Grounder re-reads sources for §11.5.2. Nil disables grounding entirely,
// which is supported: claims keep the verbatim quote checked at extraction
// time, and Grounded stays nil rather than being guessed.
Grounder *Grounder
// MaxGroundChecks bounds how many claims one grounding pass re-reads. Zero
// takes DefaultMaxGroundChecks.
MaxGroundChecks int
Log *slog.Logger
// contains filtered or unexported fields
}
Verifier builds the claim graph for a session (§11).
One pass does: read the claims nobody has scored, retrieve candidate pairs against the session's whole claim set, settle what can be settled without a model, adjudicate the rest in batches, and write the edges.
Then confidence is derived from the resulting graph (§11.3) — over the WHOLE graph, not just this pass's edges, because a new claim corroborating an existing one changes the existing claim's standing too.
func (*Verifier) Ground ¶
func (v *Verifier) Ground(ctx context.Context, sessionID string, allowance int64) (*GroundReport, error)
Ground re-reads the sources behind the claims most worth checking (§11.5.2).
allowance is the spend ceiling in the session's budget unit, normally a fraction of the escrow the caller has just released. Zero or negative disables the pass: an unbounded grounding run is exactly the failure this is designed around.
Never returns an error for a fetch that failed, a page that changed, or a model that would not answer. Those are outcomes, and recording them is the point. An error means the store or the ledger failed.
Source Files