reliability

Documentation

Overview

Package reliability measures whether a model is dependable enough to drive an agent loop, which is a different question from whether a machine can run it. It runs a fixed, version pinned battery of probes against any llm.Model and scores three capabilities that decide whether an agentic run finishes or stalls: emitting well-formed tool calls, adhering to a tool's argument schema, and following direct instructions. The result maps to a harness.ModelProfile, so a measurement made here drives how hard the harness scaffolds that model at the dispatch waist. The battery is the same regardless of which model is behind the port, so a quantized local model and a hosted one are scored on identical terms.

Index

• Constants
• func ContextCandidates(advertised int) []int
• func DescribeContext(effective, advertised int) string
• func MeasureEffectiveContext(ctx context.Context, model llm.Model, candidates []int) (int, error)
• func QuantFloor(quant string, paramsB float64) (below bool, reason string)
• type Report
  • func Score(ctx context.Context, model llm.Model) (Report, error)
  • func (r Report) Profile() harness.ModelProfile
• type Tally

Constants

const BatteryVersion = "1"

BatteryVersion pins the probe set. A score is only comparable to another taken at the same version, so it is recorded alongside the result and bumped whenever a probe changes.

Functions

func ContextCandidates

func ContextCandidates(advertised int) []int

ContextCandidates returns the token lengths to probe for a model whose advertised window is advertised: a fixed ladder of powers of two up to and including the advertised window, so the search spans the realistic range without overshooting the claim. An advertised window of zero or less yields no candidates, since there is nothing to measure against.

func DescribeContext

func DescribeContext(effective, advertised int) string

DescribeContext renders an effective-context measurement against the advertised window, for a human-facing report.

func MeasureEffectiveContext

func MeasureEffectiveContext(ctx context.Context, model llm.Model, candidates []int) (int, error)

MeasureEffectiveContext finds the largest candidate context length, in tokens, at which the model still recalls a fact buried in a prompt of that size. That is the model's usable window, which is often shorter than its advertised one, and shrinks under quantization. Candidates are tried in ascending order and the search stops at the first failure, since recall falls off with length and does not recover, so the result is the last length that passed. Zero means the model failed even the smallest candidate, so no usable window was measured. Each probe pins decoding, so the measurement is reproducible. Candidates that are not positive are ignored.

func QuantFloor

func QuantFloor(quant string, paramsB float64) (below bool, reason string)

QuantFloor judges a model+quant against the agentic-reliability floor and returns whether it is below the floor together with a human-readable reason. paramsB is the parameter count in billions (0 when unknown). The result is advisory: a quant at or above the floor still gets a caution when the model is small, and an unrecognized quant label is reported as unknown rather than silently passed, so the catalog never implies a reliability it has not checked.

Types

type Report

type Report struct {
	Version string
	Dims    map[dimension]Tally
}

Report is the outcome of running the battery: how many probes passed in each dimension, and the battery version the run used. The scores are derived from it, so the raw tallies stay available for a detailed report.

func Score

func Score(ctx context.Context, model llm.Model) (Report, error)

Score runs the full battery against model and returns the tallied report. Each probe is one model call with pinned decoding, so the measurement is reproducible on a runtime that honors the seed. A probe whose call errors counts as a failure, not a skipped probe: a model that cannot answer reliably is, for this purpose, unreliable. The context bounds the whole run, so a cancelled measurement returns its error rather than a partial score.

func (Report) Profile

func (r Report) Profile() harness.ModelProfile

Profile maps the report to the capability fingerprint the harness consumes. EffectiveContext is left zero: this battery does not probe context length, so it makes no claim about it, and the harness treats the unknown conservatively. The other three dimensions map directly.

type Tally

type Tally struct {
	Passed    int
	Attempted int
}

Tally counts the probes attempted and passed in one dimension.