armbench

package
v0.44.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Overview

Package armbench is the provenance-locked multi-arm benchmark runner (#6676, epic #6674). It replaces bespoke per-comparison scripts with ONE immutable manifest that pins every term a benchmark comparison can drift on — upstream repo/SHA/path, model snapshot, provider, sampling, max tokens, corpus hash, judge hash, trial count, seed, pairing order, concurrency, region, pricing date, and environment — and executes the declared arms against it.

The load-bearing idea is MANIFEST IDENTITY. Two benchmark runs are comparable only if the terms that decide what was measured are byte-identical; a changed model, prompt, judge, corpus, or arm capability must produce a DIFFERENT identity so a drifted rerun can never be silently stacked next to the old number. Identity is a sha256 over a canonical encoding of exactly those terms, and Selfcheck proves each of the five mutations moves it.

Everything here is pure and stdlib-only: no network, no provider SDK, no clock read except the one the caller injects. The provider and the judge are interfaces (see run.go), so the deterministic fake-provider spine that proves the runner end to end is the same code path a live provider will take.

Fail-closed is the default, not an option. A trial with no raw request or no raw response is refused (a token count with no evidence behind it is not evidence), and an arm that bundles more than one named fak capability is refused at validation (a bundled arm cannot attribute a delta to anything).

Index

Constants

View Source 
const (
	CavemanRevision = "c72984e4392c7a154e55c11dbf445f01ce5c35d4"
	CavemanModel    = "claude-sonnet-4-20250514"
)
View Source 
const (
	// FixtureImportSchema tags the machine-readable command result.
	FixtureImportSchema = "fak.armbench.fixture-import/1"

	// The exact revisions named by #6677.
	CavemanFixtureSHA  = "c72984e4392c7a154e55c11dbf445f01ce5c35d4"
	PonytailFixtureSHA = "2ed6c52c9d7e5e56942508591085fd45dea277d3"

	// CavemanLicenseReviewToken is deliberately long and revision-bound. The
	// Caveman repository uses a mixed MIT/BSL boundary and GitHub reports the
	// repository license as NOASSERTION, so importing its benchmark surfaces
	// requires the operator to state the reviewed revision and MIT conclusion.
	CavemanLicenseReviewToken = "JuliusBrussee/caveman@" + CavemanFixtureSHA + "=MIT"
)
View Source 
const (
	ReasonFixtureHashMismatch       = "FIXTURE_HASH_MISMATCH"
	ReasonFixturePathMoved          = "FIXTURE_PATH_MOVED"
	ReasonFixtureFetchFailed        = "FIXTURE_FETCH_FAILED"
	ReasonFixtureLicenseMissing     = "FIXTURE_LICENSE_METADATA_MISSING"
	ReasonFixtureLicenseReview      = "FIXTURE_LICENSE_REVIEW_REQUIRED"
	ReasonFixtureLocalMutation      = "FIXTURE_LOCAL_MUTATION"
	ReasonFixtureStoreInsideRepo    = "FIXTURE_STORE_INSIDE_REPO"
	ReasonFixtureRestrictedPath     = "FIXTURE_RESTRICTED_PATH"
	ReasonFixtureDeclarationInvalid = "FIXTURE_DECLARATION_INVALID"
)

Additional importer-specific refusal reasons. They share RefusalError with the runner so the CLI retains one exit-code/refusal-token contract.

View Source 
const (
	// ReasonManifestInvalid — a required provenance field is missing or a
	// declared value is outside its closed vocabulary.
	ReasonManifestInvalid = "MANIFEST_INVALID"
	// ReasonArmCapabilityBundled — a fak_capability arm named more than one
	// capability, so no delta it produces can be attributed to a single thing.
	ReasonArmCapabilityBundled = "ARM_CAPABILITY_BUNDLED"
	// ReasonArmCapabilityUnnamed — a fak_capability arm named no capability (or
	// a non-capability arm named one), so what it enables is unstated.
	ReasonArmCapabilityUnnamed = "ARM_CAPABILITY_UNNAMED"
	// ReasonMissingRawEvidence — a trial produced a number with no raw
	// request/response behind it.
	ReasonMissingRawEvidence = "MISSING_RAW_EVIDENCE"
	// ReasonIncomparableManifest — two runs differ on a term that decides what
	// was measured, so putting their numbers side by side would be a category
	// error.
	ReasonIncomparableManifest = "INCOMPARABLE_MANIFEST"
	// ReasonResumeIdentityMismatch — a resume ledger was produced under a
	// different manifest identity, so resuming from it would silently mix two
	// experiments.
	ReasonResumeIdentityMismatch = "RESUME_IDENTITY_MISMATCH"
	// ReasonProviderUnknown — the requested provider is not registered.
	ReasonProviderUnknown = "PROVIDER_UNKNOWN"
	// ReasonDuplicateTrial — a ledger contains the same manifest/arm/task/trial
	// key more than once, so resume or reporting would silently double count it.
	ReasonDuplicateTrial = "DUPLICATE_TRIAL"
)

The closed refusal vocabulary. Each names a distinct failure a benchmark runner must never paper over.

View Source 
const (
	PonytailRevision        = "2ed6c52c9d7e5e56942508591085fd45dea277d3"
	PonytailCavemanRevision = "c72984e4392c7a154e55c11dbf445f01ce5c35d4"
)
View Source 
const CavemanFactorialSchema = "fak-armbench-caveman-factorial/1"
View Source 
const CorpusSchema = "fak.armbench.corpus/1"

CorpusSchema tags the corpus file the runner consumes. It is a separate artifact from the manifest on purpose: the manifest pins the corpus by HASH, so the tasks themselves can be produced by the upstream fixture importer (#6677) and swapped without editing the experiment description — and any swap that is not reflected in the pinned hash shows up as a changed identity.

View Source 
const ManifestSchema = "fak.armbench.manifest/1"

ManifestSchema is the schema tag every manifest must carry. A manifest whose schema tag is absent or unknown is refused rather than best-effort parsed: this file's whole value is that a reader knows which fields were pinned.

View Source 
const PairedReceiptsSchema = "fak.armbench.paired-receipts/1"
View Source 
const PairedReportSchema = "fak.armbench.paired-report/1"
View Source 
const PassthroughSchema = "fak/armbench-caveman-passthrough/1"
View Source 
const PonytailGatesRevision = PonytailRevision
View Source 
const PonytailPromptfooRevision = "2ed6c52c9d7e5e56942508591085fd45dea277d3"
View Source 
const PonytailPromptfooVersion = "0.122.0"
View Source 
const ReportSchema = "fak.armbench.report/1"

ReportSchema tags the rolled-up report artifact.

View Source 
const RunSchema = "fak.armbench.run/1"

RunSchema tags the run artifact (the raw trial ledger + per-arm rollup).

View Source 
const SelfcheckSchema = "fak.armbench.selfcheck/1"

SelfcheckSchema tags the selfcheck artifact.

Variables

View Source 
var PonytailArms = []string{"baseline", "caveman", "ponytail"}
View Source 
var PonytailTasks = []string{
	"todo-null", "safe-path", "critic-email", "rate-limit", "sql-user", "auth-token", "csv-sum", "cache",
	"reuse-slug", "reuse-money", "trace-transfer", "trace-amount", "open-dataclass", "open-decorators",
	"open-mandelbrot", "vibe-todo", "vibe-password", "vibe-shortener", "vibe-md2html", "vibe-csvstats",
	"vibe-langgraph", "vibe-restapi", "vibe-scraper", "vibe-logparse", "vibe-rename", "vibe-adventure",
	"vibe-jsonconf", "tmpl-fe-datepicker", "tmpl-fe-colorpicker", "tmpl-fe-command", "tmpl-fe-dropzone",
	"tmpl-fe-wizard", "tmpl-fe-rating", "tmpl-be-duplicate", "tmpl-be-search", "tmpl-be-count",
	"tmpl-be-archive", "tmpl-be-bulkdelete", "tmpl-be-csv",
}

Functions

func DefaultFixtureStore 

func DefaultFixtureStore() (string, error)

DefaultFixtureStore is intentionally outside repository scratch. The caller still passes WorkspaceRoot and the importer re-checks the resolved path.

func HashTasks 

func HashTasks(tasks []Task) string

HashTasks returns the manifest corpus hash: sha256 over encoding/json's canonical encoding of the ORDERED task slice. Struct fields have a fixed order and no maps, so another Go consumer can reproduce it byte-for-byte.

func Human 

func Human(rep *Report) string

Human renders the operator-facing summary. It leads with the provenance block (a table of numbers whose pins are off-screen invites exactly the comparison this package exists to prevent), then one row per arm with input and output tokens in separate columns.

func HumanSelfcheck 

func HumanSelfcheck(r *SelfcheckResult) string

HumanSelfcheck renders the operator-facing selfcheck summary.

func MarshalCorpus 

func MarshalCorpus(c *CorpusFile) ([]byte, error)

MarshalCorpus renders a corpus file as strict, stable JSON.

func MarshalFixtureImportReport 

func MarshalFixtureImportReport(report *FixtureImportReport) ([]byte, error)

MarshalFixtureImportReport renders stable JSON for command proof.

func MarshalManifest 

func MarshalManifest(m *Manifest) ([]byte, error)

MarshalManifest renders a manifest as strict, stable JSON.

func MarshalPairedReport 

func MarshalPairedReport(r *PairedReport) ([]byte, error)

func MarshalPromptfooReproduction 

func MarshalPromptfooReproduction(r PromptfooReproduction) ([]byte, error)

func MarshalReport 

func MarshalReport(rep *Report) ([]byte, error)

MarshalReport renders the report as strict, stable JSON.

func MarshalRun 

func MarshalRun(r *Run) ([]byte, error)

MarshalRun renders a run as strict, stable JSON.

func MarshalSelfcheck 

func MarshalSelfcheck(r *SelfcheckResult) ([]byte, error)

MarshalSelfcheck renders the selfcheck artifact as strict, stable JSON.

func PonytailGateInventory 

func PonytailGateInventory(checkout string) ([]GateSource, []GateScenario, error)

func StartManagedProxy 

func StartManagedProxy(ctx context.Context, arm ManagedArm, upstream string) (baseURL string, stop func() (ProxyReceipt, error), err error)

StartManagedProxy starts an Anthropic-compatible pass-through on loopback. It never records headers or body content: only hashes and aggregate sizes.

func ValidateCorpus 

func ValidateCorpus(m *Manifest, c *CorpusFile) error

ValidateCorpus binds a loaded corpus to the manifest instead of trusting the manifest's hash label. A same-sized but edited task set is incomparable.

func WritePonytailGateReport 

func WritePonytailGateReport(path string, r PonytailGateReport) error

func WriteProxyReceipt 

func WriteProxyReceipt(path string, r ProxyReceipt) error

WriteProxyReceipt atomically writes a secret-free proxy receipt.

Types

type Arm 

type Arm struct {
	ID           string   `json:"id"`
	Kind         ArmKind  `json:"kind"`
	Capabilities []string `json:"capabilities,omitempty"`
	// PromptHash pins the system/skill prompt this arm installs. It is
	// identity-bearing: a changed prompt is a changed experiment.
	PromptHash string `json:"prompt_hash"`
	// SourceName optionally binds this arm to one of Manifest.Sources by name
	// (required for upstream_treatment — a treatment with no pinned upstream
	// input is a hand-copied approximation wearing the comparator's name).
	SourceName string `json:"source_name,omitempty"`
	Notes      string `json:"notes,omitempty"`
}

Arm is one comparison arm. Capabilities is the fail-closed field: an ArmFakCapability arm must name EXACTLY one, and every other kind must name none, so "which single thing is this arm testing" is always answerable.

type ArmKind 

type ArmKind string

ArmKind is the closed vocabulary of arm roles. The four are deliberately distinct because they answer different questions: baseline is the untreated control, upstream_treatment reproduces the comparator's own published arm, fak_passthrough charges fak's plumbing cost with NO capability enabled (the honest zero point for a fak claim), and fak_capability isolates exactly one named capability on top of that. 

const (
	// ArmBaseline is the untreated control arm.
	ArmBaseline ArmKind = "baseline"
	// ArmUpstreamTreatment reproduces the comparator's published treatment.
	ArmUpstreamTreatment ArmKind = "upstream_treatment"
	// ArmFakPassthrough routes through fak with no capability enabled — it
	// measures what fak COSTS before anything it saves is counted.
	ArmFakPassthrough ArmKind = "fak_passthrough"
	// ArmFakCapability enables exactly one named fak capability.
	ArmFakCapability ArmKind = "fak_capability"
)

func KnownArmKinds 

func KnownArmKinds() []ArmKind

KnownArmKinds returns the closed vocabulary in declaration order.

type ArmSetup 

type ArmSetup interface {
	SetupArm(ctx context.Context, arm Arm) (SetupCost, error)
}

ArmSetup is the optional half of Provider: a provider that has a real per-arm setup cost implements it and the runner charges what it reports. A provider that does not is charged zero — an honest zero, not a hidden one.

type ArmSetupCost 

type ArmSetupCost struct {
	Arm             string  `json:"arm"`
	WallMS          float64 `json:"wall_ms"`
	LocalComputeUSD float64 `json:"local_compute_usd"`
}

type ArmSummary 

type ArmSummary struct {
	ArmID        string   `json:"arm_id"`
	Kind         ArmKind  `json:"kind"`
	Capabilities []string `json:"capabilities,omitempty"`

	Trials   int `json:"trials"`
	Resumed  int `json:"resumed"`
	Failures int `json:"failures"`
	Retries  int `json:"retries"`

	// Graded is the number of trials that reached the judge. Rates below are
	// over Graded, never over Trials — dividing passes by attempted trials
	// silently converts a provider outage into a quality regression.
	Graded    int     `json:"graded"`
	Passes    int     `json:"passes"`
	PassRate  float64 `json:"pass_rate"`
	MeanScore float64 `json:"mean_score"`

	InputTokens  int     `json:"input_tokens"`
	OutputTokens int     `json:"output_tokens"`
	TotalTokens  int     `json:"total_tokens"`
	CostUSD      float64 `json:"cost_usd"`

	MeanWallMS float64 `json:"mean_wall_ms"`
	// TTFT/inter-token means are reported only over the trials that actually
	// measured them, with the count and an availability bit alongside, so an
	// unavailable timing never averages in as a zero.
	MeanTTFTMS        float64 `json:"mean_ttft_ms"`
	TTFTSamples       int     `json:"ttft_samples"`
	TTFTAvailable     bool    `json:"ttft_available"`
	MeanInterTokenMS  float64 `json:"mean_inter_token_ms"`
	InterTokenSamples int     `json:"inter_token_samples"`
	InterTokenAvail   bool    `json:"inter_token_available"`

	CacheReadTokens  int `json:"cache_read_tokens"`
	CacheWriteTokens int `json:"cache_write_tokens"`
	CacheHits        int `json:"cache_hits"`
	CacheMisses      int `json:"cache_misses"`

	Setup SetupCost `json:"setup"`
	// SetupAmortizedWallMS / SetupAmortizedCostUSD spread the one-time setup
	// across the arm's graded trials. A per-turn saving that never repays its
	// setup is not a saving, and this is the column that shows it.
	SetupAmortizedWallMS  float64 `json:"setup_amortized_wall_ms"`
	SetupAmortizedCostUSD float64 `json:"setup_amortized_cost_usd"`
}

ArmSummary is one arm's rollup. Input and output tokens stay in separate columns and the total is explicitly labelled, because an unlabelled "tokens" number is the shape a context-compression claim hides inside: an arm can cut output tokens sharply while raising input tokens, and one blended figure calls that a win.

type CacheCounters 

type CacheCounters struct {
	ReadTokens  int `json:"read_tokens"`
	WriteTokens int `json:"write_tokens"`
	Hits        int `json:"hits"`
	Misses      int `json:"misses"`
}

CacheCounters is the provider/kernel cache accounting for one trial.

type CavemanCall 

type CavemanCall struct {
	PromptID, Arm      string
	Trial              int
	Text, FinishReason string
	Usage              CavemanUsage
	SemanticPass       bool
	Missing            []string `json:",omitempty"`
	Raw                json.RawMessage
}

type CavemanOptions 

type CavemanOptions struct {
	InputDir, OutDir, BaseURL, APIKey, Model, Label string
	Trials                                          int
}

type CavemanPacket 

type CavemanPacket struct {
	Schema, Source, Revision, RunLabel, ProviderEndpoint, RequestedModel, ResolvedModel string
	ExactModel                                                                          bool
	Temperature                                                                         int
	MaxOutputTokens, Trials                                                             int
	Hashes                                                                              map[string]string
	Calls                                                                               []CavemanCall
	Summary                                                                             []CavemanSummary
	Upstream                                                                            map[string]any
	GeneratedAt                                                                         string
}

func RunCaveman 

func RunCaveman(ctx context.Context, o CavemanOptions) (CavemanPacket, error)

type CavemanPrompt 

type CavemanPrompt struct{ ID, Category, Prompt string }

type CavemanSummary 

type CavemanSummary struct {
	Arm                           string
	MedianOutputByPrompt          map[string]int
	AverageMedian                 float64
	SemanticPassed, SemanticTotal int
}

type CavemanUsage 

type CavemanUsage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

type Check 

type Check struct {
	Name     string `json:"name"`
	OK       bool   `json:"ok"`
	Detail   string `json:"detail"`
	Evidence string `json:"evidence,omitempty"`
}

Check is one selfcheck assertion and its captured evidence.

type ClaimCheckInput 

type ClaimCheckInput struct {
	Claim         string `json:"claim"`
	TunedBaseline string `json:"tuned_baseline"`
	Scope         string `json:"scope"`
	Provenance    string `json:"provenance"`
	Witness       string `json:"witness"`
	Verdict       string `json:"verdict"`
}

type ComparabilityField 

type ComparabilityField struct {
	Field string `json:"field"`
	A     string `json:"a"`
	B     string `json:"b"`
}

ComparabilityField names one term two manifests disagreed on.

func CheckComparable 

func CheckComparable(a, b *Manifest) ([]ComparabilityField, error)

CheckComparable refuses two manifests whose reports must not be treated as repeat measurements of one experiment. Arms are compared WITHIN one run; a changed arm contract, model, corpus, schedule, or environment is a different manifest and the returned field list makes that drift explicit.

It reports every disagreeing field rather than the first, because the operator fixing a drifted manifest wants the whole list in one pass.

type Corpus 

type Corpus struct {
	ID        string `json:"id"`
	Hash      string `json:"hash"`
	TaskCount int    `json:"task_count"`
}

Corpus pins the task set by content hash. TaskCount is recorded so a truncated corpus is visible in the report even before the hash is recomputed.

type CorpusFile 

type CorpusFile struct {
	Schema string `json:"schema"`
	ID     string `json:"id"`
	Tasks  []Task `json:"tasks"`
}

CorpusFile is the on-disk corpus.

func DemoCorpusFile 

func DemoCorpusFile() *CorpusFile

DemoCorpusFile is the demo corpus in its on-disk shape, so `--emit-demo` writes a pair an operator can run unedited.

func UnmarshalCorpus 

func UnmarshalCorpus(b []byte) (*CorpusFile, error)

UnmarshalCorpus parses a corpus file, refusing an unknown schema tag or an unknown field rather than silently measuring a shape it does not understand.

type CostSensitivity 

type CostSensitivity struct {
	Scenario             string   `json:"scenario"`
	ProviderBaselineUSD  float64  `json:"provider_baseline_usd"`
	ProviderTreatmentUSD float64  `json:"provider_treatment_usd"`
	SteadyDeltaUSD       Interval `json:"steady_state_paired_delta_usd"`
	SetupDeltaUSD        float64  `json:"one_time_setup_delta_usd"`
	SetupDeltaMS         float64  `json:"one_time_setup_delta_ms"`
	Amortized100DeltaUSD float64  `json:"amortized_delta_usd_at_100_trials"`
	BreakEvenTrials      *float64 `json:"break_even_trials,omitempty"`
}

type Environment 

type Environment struct {
	OS          string `json:"os"`
	Arch        string `json:"arch"`
	HostClass   string `json:"host_class"`
	FakVersion  string `json:"fak_version"`
	PricingDate string `json:"pricing_date"`
}

Environment records the host and pricing context. It is identity-bearing so a changed machine or price sheet cannot hide behind the old manifest id; a later comparison verb (#6680) can still classify the named differing fields.

type FactorialCell 

type FactorialCell struct {
	Style                    string             `json:"style"`
	Treatment                FactorialTreatment `json:"treatment"`
	Workload                 string             `json:"workload"`
	Pressure                 int                `json:"pressure"`
	Turns                    int                `json:"turns"`
	OutputTokensEstimated    int                `json:"answer_output_tokens_estimated"`
	OutputTokensProvider     *int64             `json:"answer_output_tokens_provider"`
	ProviderOutput           string             `json:"provider_output,omitempty"`
	ProviderInputTokens      *int64             `json:"provider_input_tokens"`
	ProviderCacheReadTokens  *int64             `json:"provider_cache_read_tokens"`
	ProviderCacheWriteTokens *int64             `json:"provider_cache_write_tokens"`
	RetainedFacts            int                `json:"retained_facts"`
	TotalFacts               int                `json:"total_facts"`
	Quality                  float64            `json:"quality"`
	Stages                   []FactorialStage   `json:"stages"`
	// contains filtered or unexported fields
}

type FactorialManifest 

type FactorialManifest struct {
	Schema           string           `json:"schema"`
	Comparator       string           `json:"comparator"`
	EvidenceClass    string           `json:"evidence_class"`
	TokenMethod      string           `json:"token_method"`
	DisabledFeatures []string         `json:"disabled_features"`
	ProviderEndpoint string           `json:"provider_endpoint,omitempty"`
	ProviderModel    string           `json:"provider_model,omitempty"`
	Pressures        []int            `json:"pressures"`
	Cells            []FactorialCell  `json:"cells"`
	QualityFrontier  []FrontierPoint  `json:"quality_frontier"`
	Interactions     []InteractionRow `json:"interaction_effects"`
	Conclusion       string           `json:"conclusion"`
}

func RunCavemanFactorial 

func RunCavemanFactorial(o FactorialOptions) (FactorialManifest, error)

type FactorialOptions 

type FactorialOptions struct {
	OutputDir string
	Pressures []int
	BaseURL   string
	APIKey    string
	Model     string
	InputDir  string
}

type FactorialStage 

type FactorialStage struct {
	Name         string `json:"name"`
	CPUTimeNS    int64  `json:"cpu_time_ns"`
	BytesBefore  int    `json:"bytes_before"`
	BytesAfter   int    `json:"bytes_after"`
	TokensBefore int    `json:"tokens_before_estimated"`
	TokensAfter  int    `json:"tokens_after_estimated"`
}

type FactorialTreatment 

type FactorialTreatment string
const (
	TreatmentPassthrough FactorialTreatment = "passthrough"
	TreatmentCompress    FactorialTreatment = "tool-result-compression"
	TreatmentShed        FactorialTreatment = "context-shedding"
	TreatmentBoth        FactorialTreatment = "compression+shedding"
	TreatmentTuned       FactorialTreatment = "tuned-bundle"
)

type FakeGrader 

type FakeGrader struct {
	// FailArm, when non-empty, marks every trial of that arm as failing the
	// judge. It is the fixture that proves a token saving with a correctness
	// loss is visible in the rollup rather than absorbed into the mean.
	FailArm string
}

FakeGrader is the deterministic spine grader. It grades on the synthetic answer's shape and always records its raw judgment, so the evidence fence has something real to check.

func (*FakeGrader) Grade 

func (g *FakeGrader) Grade(_ context.Context, req Request, resp Response) (Judgment, error)

Grade returns the deterministic verdict for one completed trial.

type FakeProvider 

type FakeProvider struct {
	// SetupWallMS is charged once per non-baseline arm. Zero means the arms are
	// setup-free, which is recorded as an honest zero.
	SetupWallMS float64
	// OmitRawResponse, when non-empty, makes the provider return an empty raw
	// response for that arm id. It is the negative fixture for the
	// MISSING_RAW_EVIDENCE fence — a provider that reports usage with nothing
	// behind it.
	OmitRawResponse string
}

FakeProvider is the deterministic spine provider. It exists so the whole runner — pairing, ordering, resume, evidence fencing, rollup — can be proven end to end with no network, no key, and no model, and so that the exact code path a live provider will take is the one already under test.

Determinism is total: every number it returns is a pure function of (arm id, arm kind, capabilities, task id, trial, prompt hash, model snapshot). Two runs of the same manifest therefore produce byte-identical ledgers, which is what makes the resume and identity proofs checkable rather than flaky.

The shape of the synthetic numbers is deliberately NOT flattering: the fak_passthrough arm costs a little more wall time than baseline (plumbing is not free), and the capability arm's saving is on OUTPUT tokens while its input tokens rise slightly — the exact pattern a blended token column would hide.

func (*FakeProvider) Complete 

func (p *FakeProvider) Complete(_ context.Context, req Request) (Response, error)

Complete returns the deterministic synthetic trial.

func (*FakeProvider) SetupArm 

func (p *FakeProvider) SetupArm(_ context.Context, arm Arm) (SetupCost, error)

SetupArm charges the declared one-time setup to every arm that installs a treatment; the untreated baseline installs nothing and pays nothing.

type FixtureDeclaration 

type FixtureDeclaration struct {
	Name                string
	Suite               FixtureSuite
	Repo                string
	SHA                 string
	Path                string
	ExpectedHash        string
	Role                string
	License             string
	LicenseBoundary     string
	LicenseBoundaryHash string
	ReviewToken         string
	Normalization       string
}

FixtureDeclaration is one reviewed upstream path. ExpectedHash is over the raw response body, before any newline or text normalization (there is none).

func PinnedFixtureDeclarations 

func PinnedFixtureDeclarations(suite FixtureSuite) []FixtureDeclaration

PinnedFixtureDeclarations returns a copy of the audited source allowlist.

type FixtureImportReport 

type FixtureImportReport struct {
	Schema      string                `json:"schema"`
	RetrievedAt string                `json:"retrieved_at"`
	Store       string                `json:"store"`
	Results     []FixtureImportResult `json:"results"`
}

FixtureImportReport is the stable JSON shape emitted by the command.

func ImportFixtures 

func ImportFixtures(ctx context.Context, suite FixtureSuite, opts ImportOptions) (*FixtureImportReport, error)

ImportFixtures materializes one or both pinned suites.

type FixtureImportResult 

type FixtureImportResult struct {
	Suite            FixtureSuite    `json:"suite"`
	InputID          string          `json:"input_id"`
	InputDir         string          `json:"input_dir"`
	ManifestPath     string          `json:"manifest_path"`
	CorpusPath       string          `json:"corpus_path"`
	ManifestIdentity string          `json:"manifest_identity"`
	SourceSetHash    string          `json:"source_set_hash"`
	SourceCount      int             `json:"source_count"`
	SourceBytes      int64           `json:"source_bytes"`
	LicenseReviews   []LicenseReview `json:"license_reviews"`
}

FixtureImportResult is one suite's content-addressed armbench input. Paths are relative to StoreRoot so committed proof output contains no host path.

type FixtureSuite 

type FixtureSuite string

FixtureSuite is the closed importer suite vocabulary. 

const (
	FixtureSuiteCaveman  FixtureSuite = "caveman"
	FixtureSuitePonytail FixtureSuite = "ponytail"
	FixtureSuiteAll      FixtureSuite = "all"
)

type FrontierPoint 

type FrontierPoint struct {
	Style                 string             `json:"style"`
	Treatment             FactorialTreatment `json:"treatment"`
	Workload              string             `json:"workload"`
	Pressure              int                `json:"pressure"`
	InputBytes            int                `json:"input_bytes"`
	OutputTokensEstimated int                `json:"output_tokens_estimated"`
	Quality               float64            `json:"quality"`
}

type GateCell 

type GateCell struct {
	ScenarioID string `json:"scenario_id"`
	Arm        string `json:"arm"`
	Category   string `json:"category"`
	Pass       bool   `json:"pass"`
	Reason     string `json:"reason"`
	Output     string `json:"output,omitempty"`
	DurationMS int64  `json:"duration_ms,omitempty"`
	Error      string `json:"error,omitempty"`
}

type GateRunArtifact 

type GateRunArtifact struct {
	ID      string `json:"id"`
	Command string `json:"command"`
	Pass    bool   `json:"pass"`
	Output  string `json:"output"`
	Error   string `json:"error,omitempty"`
}

type GateScenario 

type GateScenario struct {
	ID               string `json:"id"`
	Set              string `json:"set"`
	Category         string `json:"category"`
	Task             string `json:"task"`
	SourcePath       string `json:"source_path"`
	SourceSHA256     string `json:"source_sha256"`
	Assertion        string `json:"assertion"`
	RequiresProvider bool   `json:"requires_provider"`
	Exclusion        string `json:"exclusion,omitempty"`
}

type GateSource 

type GateSource struct {
	Path   string `json:"path"`
	SHA256 string `json:"sha256"`
	Bytes  int    `json:"bytes"`
}

type GateSummary 

type GateSummary struct {
	Arm      string `json:"arm"`
	Category string `json:"category"`
	Passed   int    `json:"passed"`
	Failed   int    `json:"failed"`
	NotRun   int    `json:"not_run"`
	GatePass bool   `json:"gate_pass"`
}

type Grader 

type Grader interface {
	Grade(ctx context.Context, req Request, resp Response) (Judgment, error)
}

Grader grades one completed trial.

type ImportOptions 

type ImportOptions struct {
	StoreRoot      string
	WorkspaceRoot  string
	BaseURL        string
	FakVersion     string
	Client         *http.Client
	Now            func() time.Time
	LicenseReviews []string
}

ImportOptions contains the impure edges. Tests inject an httptest server, clock, and temp roots, so CI never needs the network.

type InteractionRow 

type InteractionRow struct {
	Workload        string             `json:"workload"`
	Pressure        int                `json:"pressure"`
	Treatment       FactorialTreatment `json:"treatment"`
	InputBytesDID   int                `json:"input_bytes_difference_in_differences"`
	OutputTokensDID int                `json:"output_tokens_difference_in_differences_estimated"`
	QualityDID      float64            `json:"quality_difference_in_differences"`
	Classification  string             `json:"classification"`
}

type Interval 

type Interval struct {
	Estimate   float64 `json:"estimate"`
	Lower      float64 `json:"lower"`
	Upper      float64 `json:"upper"`
	Confidence float64 `json:"confidence"`
}

type Judge 

type Judge struct {
	ID   string `json:"id"`
	Hash string `json:"hash"`
	Kind string `json:"kind,omitempty"`
}

Judge pins the grader: its identifier and the content hash of its definition (prompt, rubric, or deterministic checker source). A changed judge changes the manifest identity, because a score graded by a different judge is a different measurement.

type Judgment 

type Judgment struct {
	Pass        bool    `json:"pass"`
	Score       float64 `json:"score"`
	Reason      string  `json:"reason,omitempty"`
	RawJudgment string  `json:"raw_judgment"`
}

Judgment is the grader's verdict for one trial. RawJudgment is the judge's own evidence and is required for a graded trial, for the same reason the provider owes a raw response.

type Latency 

type Latency struct {
	WallMS              float64 `json:"wall_ms"`
	TTFTMS              float64 `json:"ttft_ms"`
	TTFTAvailable       bool    `json:"ttft_available"`
	InterTokenMS        float64 `json:"inter_token_ms"`
	InterTokenAvailable bool    `json:"inter_token_available"`
}

Latency separates the three timings a streaming provider can report, each with an explicit availability bit. A provider that cannot measure TTFT reports TTFTAvailable=false rather than 0, so the report can say "unavailable" instead of averaging zeros into a fictional speedup.

type LicenseReview 

type LicenseReview struct {
	Repo     string `json:"repo"`
	SHA      string `json:"sha"`
	License  string `json:"license"`
	Boundary string `json:"boundary"`
	Status   string `json:"status"`
}

LicenseReview records how one repository's license boundary was admitted.

type ManagedArm 

type ManagedArm string

ManagedArm is one explicitly isolated fak context treatment. These values are serialized into receipts; do not rename them without a schema revision. 

const (
	ManagedDirect        ManagedArm = "direct"
	ManagedPassthrough   ManagedArm = "fak_passthrough"
	ManagedProviderCache ManagedArm = "shared_prefix_provider_cache_only"
	ManagedCompression   ManagedArm = "tool_result_compression_only"
	ManagedShedding      ManagedArm = "context_shedding_only"
	ManagedBundle        ManagedArm = "compression_shedding_bundle"
)

func ManagedArms 

func ManagedArms() []ManagedArm

type ManagedRunOptions 

type ManagedRunOptions struct {
	UpstreamDir string
	Tasks       []string
	Treatments  []string
	Arms        []ManagedArm
	Model       string
	Runs        int
	Workers     int
	DryRun      bool
	ReceiptPath string
	UpstreamURL string
}

type ManagedRunReceipt 

type ManagedRunReceipt struct {
	Schema                   string         `json:"schema"`
	ProviderBacked           bool           `json:"provider_backed"`
	Comparator               string         `json:"comparator"`
	UpstreamGitHead          string         `json:"upstream_git_head"`
	Treatments               []string       `json:"treatments"`
	ManagedArms              []ManagedArm   `json:"managed_arms"`
	Tasks                    []string       `json:"tasks"`
	Model                    string         `json:"model"`
	Runs                     int            `json:"runs"`
	Workers                  int            `json:"workers"`
	AccountIdentity          string         `json:"account_identity"`
	CredentialValuesRecorded bool           `json:"credential_values_recorded"`
	ExcludedFeatures         ManagedToggles `json:"excluded_features"`
	Commands                 []string       `json:"commands"`
	RunDirs                  []string       `json:"run_dirs,omitempty"`
	ProxyReceipts            []ProxyReceipt `json:"proxy_receipts,omitempty"`
}

func RunManagedMatrix 

func RunManagedMatrix(ctx context.Context, o ManagedRunOptions) (ManagedRunReceipt, error)

type ManagedToggles 

type ManagedToggles struct {
	FakPath                   bool `json:"fak_path"`
	SharedPrefixProviderCache bool `json:"shared_prefix_provider_cache"`
	ToolResultCompression     bool `json:"tool_result_compression"`
	ContextShedding           bool `json:"context_shedding"`
	Routing                   bool `json:"routing"`
	Policy                    bool `json:"policy"`
	ResponseReuse             bool `json:"response_reuse"`
}

ManagedToggles is the auditable switchboard written beside every run.

func TogglesForManagedArm 

func TogglesForManagedArm(a ManagedArm) (ManagedToggles, error)

type Manifest 

type Manifest struct {
	Schema      string      `json:"schema"`
	ID          string      `json:"id"`
	Sources     []Source    `json:"sources"`
	Model       Model       `json:"model"`
	Corpus      Corpus      `json:"corpus"`
	Judge       Judge       `json:"judge"`
	Trials      Trials      `json:"trials"`
	Environment Environment `json:"environment"`
	Arms        []Arm       `json:"arms"`
}

Manifest is the immutable description of one multi-arm comparison.

func DemoManifest 

func DemoManifest() *Manifest

DemoManifest is the pinned four-arm demo: an untreated baseline, the upstream Caveman treatment at the SHA epic #6674 pins, fak passthrough, and exactly one isolated fak capability. It is the shape every later benchmark issue instantiates, so it doubles as the worked example in the docs.

func UnmarshalManifest 

func UnmarshalManifest(b []byte) (*Manifest, error)

UnmarshalManifest parses a manifest, refusing unknown fields so a typo'd provenance key is a refusal rather than a silently-unpinned term.

func (*Manifest) ArmByID 

func (m *Manifest) ArmByID(id string) (Arm, bool)

ArmByID returns the declared arm with the given id.

func (*Manifest) Identity 

func (m *Manifest) Identity() string

Identity is the sha256 over the canonical encoding of the identity-bearing terms. Two manifests share an identity exactly when they describe the same measurement; a changed model, prompt, judge, corpus, or arm capability moves it (proven by Selfcheck). Scheduling and environment are included too because they can move measured latency/cost even when prompts and model stay fixed.

func (*Manifest) Validate 

func (m *Manifest) Validate() error

Validate refuses a manifest that is not fully pinned. Every check here exists because the missing field makes a published number unreproducible or unattributable; there is no "warn" tier.

type MetricComparison 

type MetricComparison struct {
	Metric        string   `json:"metric"`
	Unit          string   `json:"unit"`
	BaselineMean  float64  `json:"baseline_mean"`
	TreatmentMean float64  `json:"treatment_mean"`
	Delta         Interval `json:"paired_delta_treatment_minus_baseline"`
}

type Model 

type Model struct {
	Provider  string   `json:"provider"`
	Snapshot  string   `json:"snapshot"`
	Region    string   `json:"region"`
	Sampling  Sampling `json:"sampling"`
	MaxTokens int      `json:"max_tokens"`
}

Model pins the generation side: which provider, which model SNAPSHOT (never a floating alias — an alias silently repoints and destroys comparability), the region it was served from, the sampling parameters, and the output cap.

type Options 

type Options struct {
	// Resume, when non-nil, supplies a prior run whose completed trials are
	// carried over instead of re-executed.
	Resume *Run
}

Options configures one Run call.

type OrderStrategy 

type OrderStrategy string

OrderStrategy decides the within-pair execution order of the arms for one (task, trial) unit. Both settings are deterministic given the manifest seed — "randomized" means randomized ACROSS pairs, not irreproducible. 

const (
	// OrderCounterbalanced rotates the arm order by (task index + trial), so
	// every arm occupies every position an equal number of times.
	OrderCounterbalanced OrderStrategy = "counterbalanced"
	// OrderRandomized shuffles the arm order with a seeded PRNG derived from
	// (seed, task id, trial), so the order is random across pairs and exactly
	// reproducible from the manifest.
	OrderRandomized OrderStrategy = "randomized"
)

type PairUnit 

type PairUnit struct {
	TaskIndex int
	TaskID    string
	Trial     int
	ArmOrder  []string
}

PairUnit is one paired execution unit: the same (task, trial) run through EVERY arm, back to back, in the unit's counterbalanced or seeded-random arm order. Pairing at this granularity is what makes the comparison paired — arms see the same task under the same conditions, and no arm is systematically advantaged by always going first (a warm-cache and a rate-limit artifact both favour position).

func PlanUnits 

func PlanUnits(m *Manifest, tasks []Task) []PairUnit

PlanUnits derives the deterministic execution plan from the manifest and the corpus. It is exported and pure so the ordering can be asserted in a test without running a provider.

type PairedComparison 

type PairedComparison struct {
	Task                    string             `json:"task"`
	Model                   string             `json:"model"`
	Temperature             string             `json:"temperature"`
	Treatment               string             `json:"treatment"`
	Pairs                   int                `json:"pairs"`
	ColdPairs               int                `json:"cold_pairs"`
	WarmPairs               int                `json:"warm_pairs"`
	BaselineFailures        int                `json:"baseline_failures"`
	TreatmentFailures       int                `json:"treatment_failures"`
	Correctness             Interval           `json:"success_rate_paired_delta"`
	Safety                  Interval           `json:"safety_rate_paired_delta"`
	CorrectnessGate         string             `json:"correctness_gate"`
	SafetyGate              string             `json:"safety_gate"`
	EfficiencyClaimsAllowed bool               `json:"efficiency_claims_allowed"`
	Headline                string             `json:"headline"`
	Metrics                 []MetricComparison `json:"metrics"`
	ColdMetrics             []MetricComparison `json:"cold_metrics"`
	WarmMetrics             []MetricComparison `json:"warm_metrics"`
	Costs                   []CostSensitivity  `json:"cost_sensitivity"`
}

type PairedReceipts 

type PairedReceipts struct {
	Schema            string          `json:"schema"`
	Benchmark         string          `json:"benchmark"`
	TunedBaseline     string          `json:"tuned_baseline"`
	CorrectnessMargin float64         `json:"correctness_noninferiority_margin"`
	SafetyMargin      float64         `json:"safety_noninferiority_margin"`
	BootstrapSamples  int             `json:"bootstrap_samples,omitempty"`
	Provenance        string          `json:"provenance"`
	Witness           string          `json:"witness"`
	Setup             []ArmSetupCost  `json:"setup"`
	Prices            []PriceScenario `json:"price_scenarios"`
	Trials            []PairedTrial   `json:"trials"`
}

func UnmarshalPairedReceipts 

func UnmarshalPairedReceipts(b []byte) (*PairedReceipts, error)

type PairedReport 

type PairedReport struct {
	Schema          string             `json:"schema"`
	Benchmark       string             `json:"benchmark"`
	TunedBaseline   string             `json:"tuned_baseline"`
	Confidence      float64            `json:"confidence"`
	FamilywiseAlpha float64            `json:"familywise_alpha"`
	Correction      string             `json:"multiple_comparison_correction"`
	Comparisons     []PairedComparison `json:"comparisons"`
	ClaimCheck      []ClaimCheckInput  `json:"claim_check_input"`
}

func BuildPairedReport 

func BuildPairedReport(in *PairedReceipts) (*PairedReport, error)

type PairedTrial 

type PairedTrial struct {
	PairID       string  `json:"pair_id"`
	Task         string  `json:"task"`
	Model        string  `json:"model"`
	Temperature  string  `json:"temperature"`
	Arm          string  `json:"arm"`
	Success      bool    `json:"success"`
	Safe         bool    `json:"safe"`
	InputTokens  float64 `json:"input_tokens"`
	OutputTokens float64 `json:"output_tokens"`
	WallMS       float64 `json:"wall_ms"`
	TTFTMS       float64 `json:"ttft_ms"`
	Retries      float64 `json:"retries"`
	Failed       bool    `json:"failed"`
	Cold         bool    `json:"cold"`
}

type PassthroughAggregate 

type PassthroughAggregate struct {
	Calls, SemanticPassed, Input, Output, CacheWrite, CacheRead int
	MedianTTFTMS, MedianWallMS, MedianFakOverheadMS             float64
	CostUSD                                                     *float64
}

type PassthroughCall 

type PassthroughCall struct {
	PromptID, Arm, Phase, Text, FinishReason string
	Trial                                    int
	TTFTMS, WallMS, FakOverheadMS            float64
	Usage                                    TokenEvidence
	CostUSD                                  *float64
	SemanticPass                             bool
	SemanticMissing                          []string
	Request                                  json.RawMessage
	RawSSE                                   string
}

type PassthroughManifest 

type PassthroughManifest struct {
	Schema, Source, Revision, RunLabel, ProviderEndpoint, Model string
	Trials                                                      int
	ExactModel                                                  bool
	Features                                                    map[string]any
	Hashes                                                      map[string]string
	Calls                                                       []PassthroughCall
	Summary                                                     []PassthroughSummary
	CacheVerdict, CostVerdict, Conclusion                       string
}

func RunCavemanPassthrough 

func RunCavemanPassthrough(ctx context.Context, o PassthroughOptions) (PassthroughManifest, error)

type PassthroughOptions 

type PassthroughOptions struct {
	InputDir, OutDir, BaseURL, APIKey, Model, Label        string
	Trials                                                 int
	InputPerMillion, OutputPerMillion, CacheReadPerMillion float64
}

type PassthroughSummary 

type PassthroughSummary struct {
	Arm        string
	Cold, Warm PassthroughAggregate
}

type PonytailArmReport 

type PonytailArmReport struct {
	Arm       string  `json:"arm"`
	Successes int     `json:"task_successes"`
	Safe      int     `json:"safe"`
	Cells     int     `json:"cells"`
	Tokens    int64   `json:"tokens"`
	CostUSD   float64 `json:"cost_usd"`
	LatencyMS int64   `json:"latency_ms"`
	Denials   int     `json:"permission_denials"`
	Failures  int     `json:"failures"`
	Retries   int     `json:"retries"`
}

func SummarizePonytailEvidence 

func SummarizePonytailEvidence(root string) ([]PonytailArmReport, error)

SummarizePonytailEvidence folds unchanged upstream results.json files. Upstream has no cell retry loop, so retries are explicitly zero; process errors and missing correctness become failures.

type PonytailGateOptions 

type PonytailGateOptions struct {
	Checkout, Claude, Model, Account, Replay string
	Live                                     bool
	Trials                                   int
	Timeout                                  time.Duration
}

type PonytailGateReport 

type PonytailGateReport struct {
	Schema            string            `json:"schema"`
	Comparator        string            `json:"comparator"`
	GeneratedAt       string            `json:"generated_at"`
	Live              bool              `json:"live"`
	Model             string            `json:"model,omitempty"`
	Account           string            `json:"account,omitempty"`
	Trials            int               `json:"trials"`
	Sources           []GateSource      `json:"sources"`
	Scenarios         []GateScenario    `json:"scenarios"`
	Cells             []GateCell        `json:"cells"`
	Summary           []GateSummary     `json:"summary"`
	DeterministicRuns []GateRunArtifact `json:"deterministic_runs"`
	OverallPass       bool              `json:"overall_pass"`
	Assumptions       []string          `json:"assumptions"`
	Extensions        []GateCell        `json:"extensions"`
}

func RunPonytailGates 

func RunPonytailGates(ctx context.Context, o PonytailGateOptions) (PonytailGateReport, error)

type PonytailOptions 

type PonytailOptions struct {
	Checkout, Caveman, Out, Account, Python, Model string
	Trials                                         int
	Live                                           bool
}

type PonytailPacket 

type PonytailPacket struct {
	Schema         string              `json:"schema"`
	Mode           string              `json:"mode"`
	Comparator     string              `json:"comparator"`
	Revision       string              `json:"revision"`
	Caveman        string              `json:"caveman,omitempty"`
	Checkout       string              `json:"checkout"`
	Files          map[string]string   `json:"files_sha256"`
	Tasks          []string            `json:"tasks"`
	Arms           []string            `json:"arms"`
	Models         map[string]string   `json:"models"`
	AgentModel     string              `json:"agent_model"`
	JudgeModel     string              `json:"judge_model"`
	TimeoutSeconds int                 `json:"timeout_seconds"`
	Trials         int                 `json:"trials"`
	Workers        int                 `json:"workers"`
	Account        string              `json:"account_identity,omitempty"`
	Counterbalance [][]string          `json:"counterbalanced_orders"`
	Commands       []string            `json:"commands"`
	StartedAt      string              `json:"started_at,omitempty"`
	FinishedAt     string              `json:"finished_at,omitempty"`
	Runs           []PonytailRun       `json:"runs,omitempty"`
	Report         []PonytailArmReport `json:"report_task_success_first,omitempty"`
}

PonytailPacket is both the no-spend launch packet and the live run receipt.

func Ponytail 

func Ponytail(opts PonytailOptions) (PonytailPacket, error)

type PonytailRun 

type PonytailRun struct {
	Task       string   `json:"task"`
	Order      []string `json:"order"`
	OutputDir  string   `json:"output_dir"`
	ExitCode   int      `json:"exit_code"`
	DurationMS int64    `json:"duration_ms"`
	Stdout     string   `json:"stdout"`
	Stderr     string   `json:"stderr"`
}

type PriceScenario 

type PriceScenario struct {
	Name                   string  `json:"name"`
	InputUSDPerMillion     float64 `json:"input_usd_per_million"`
	OutputUSDPerMillion    float64 `json:"output_usd_per_million"`
	LocalComputeMultiplier float64 `json:"local_compute_multiplier"`
}

type PromptfooCell 

type PromptfooCell struct {
	Config     string   `json:"config"`
	Provider   string   `json:"provider"`
	Arms       []string `json:"arms"`
	Status     string   `json:"status"`
	ExitCode   int      `json:"exit_code"`
	ResultPath string   `json:"result_path,omitempty"`
	StdoutPath string   `json:"stdout_path"`
	StderrPath string   `json:"stderr_path"`
	Attempts   int      `json:"attempts"`
	Detail     string   `json:"detail,omitempty"`
	Command    []string `json:"command,omitempty"`
}

type PromptfooInput 

type PromptfooInput struct {
	Path   string `json:"path"`
	Kind   string `json:"kind"`
	SHA256 string `json:"sha256"`
	Bytes  int64  `json:"bytes"`
}

type PromptfooReproduction 

type PromptfooReproduction struct {
	Schema           string           `json:"schema"`
	Upstream         string           `json:"upstream"`
	Revision         string           `json:"revision"`
	PromptfooVersion string           `json:"promptfoo_version"`
	CapturedAt       string           `json:"captured_at"`
	Inputs           []PromptfooInput `json:"inputs"`
	Cells            []PromptfooCell  `json:"cells"`
	ValueClaim       string           `json:"fak_value_claim"`
	Complete         bool             `json:"all_declared_cells_attempted"`
}

func RunPonytailPromptfoo 

func RunPonytailPromptfoo(source, outDir string, execute bool) (PromptfooReproduction, error)

type Provider 

type Provider interface {
	Complete(ctx context.Context, req Request) (Response, error)
}

Provider executes one trial for one arm.

type ProxyReceipt 

type ProxyReceipt struct {
	Schema                  string         `json:"schema"`
	Arm                     ManagedArm     `json:"arm"`
	Toggles                 ManagedToggles `json:"toggles"`
	Requests                int64          `json:"requests"`
	InputBytes              int64          `json:"input_bytes"`
	RetainedContextBytes    int64          `json:"retained_context_bytes"`
	OutputBytes             int64          `json:"output_bytes"`
	CacheControlWrites      int64          `json:"cache_control_writes"`
	CompressedToolResults   int64          `json:"compressed_tool_results"`
	ShedMessages            int64          `json:"shed_messages"`
	TransformCPUNanoseconds int64          `json:"fak_cpu_nanoseconds"`
	TTFTMilliseconds        []int64        `json:"ttft_ms"`
	WallMilliseconds        []int64        `json:"wall_ms"`
	RequestSHA256           []string       `json:"request_sha256"`
}

type RefusalError 

type RefusalError struct {
	Reason string
	Detail string
}

RefusalError is a typed refusal carrying a token from a closed vocabulary, so a caller can branch on the reason instead of matching prose.

func (*RefusalError) Error 

func (e *RefusalError) Error() string

type Report 

type Report struct {
	Schema           string       `json:"schema"`
	ManifestIdentity string       `json:"manifest_identity"`
	ManifestID       string       `json:"manifest_id"`
	Model            Model        `json:"model"`
	Corpus           Corpus       `json:"corpus"`
	Judge            Judge        `json:"judge"`
	Trials           Trials       `json:"trials"`
	Environment      Environment  `json:"environment"`
	Sources          []Source     `json:"sources"`
	Arms             []ArmSummary `json:"arms"`

	TotalTrials   int `json:"total_trials"`
	ExecutedCount int `json:"executed"`
	ResumedCount  int `json:"resumed"`
	FailureCount  int `json:"failures"`
}

Report is the rolled-up, publishable view of one run.

func Summarize 

func Summarize(r *Run) (*Report, error)

Summarize folds a raw run into per-arm rollups. It re-checks the evidence fence on every row: a run artifact can arrive from disk, and a report is the thing that gets published, so the last gate before publication re-asks the question rather than trusting that the producer asked it.

type Request 

type Request struct {
	ManifestIdentity string   `json:"manifest_identity"`
	ArmID            string   `json:"arm_id"`
	ArmKind          ArmKind  `json:"arm_kind"`
	Capabilities     []string `json:"capabilities,omitempty"`
	TaskID           string   `json:"task_id"`
	Trial            int      `json:"trial"`
	Position         int      `json:"position"`
	Input            string   `json:"input"`
	PromptHash       string   `json:"prompt_hash"`
	Model            Model    `json:"model"`
}

Request is exactly what an arm asks the provider for one trial. It carries the arm's identity so a provider can install the arm's treatment, and the trial index so a provider that seeds per-trial can be reproducible.

type Response 

type Response struct {
	RawRequest  string        `json:"raw_request"`
	RawResponse string        `json:"raw_response"`
	Text        string        `json:"text"`
	Usage       Usage         `json:"usage"`
	Latency     Latency       `json:"latency"`
	Cache       CacheCounters `json:"cache"`
	Retries     int           `json:"retries"`
	// Failure, when non-empty, marks a trial the provider could not complete.
	// A failed trial still owes a raw request and a reason — it is counted in
	// the report's failure column, never dropped.
	Failure string `json:"failure,omitempty"`
}

Response is what a provider returns for one trial. RawRequest and RawResponse are the evidence: the runner refuses a trial that reports usage or latency without them (see checkEvidence).

type Run 

type Run struct {
	Schema           string               `json:"schema"`
	ManifestIdentity string               `json:"manifest_identity"`
	Manifest         *Manifest            `json:"manifest"`
	Setup            map[string]SetupCost `json:"setup"`
	Trials           []TrialResult        `json:"trials"`
	Executed         int                  `json:"executed"`
	ResumedCount     int                  `json:"resumed"`
}

Run is the run artifact: the manifest it came from, its identity, every raw trial row, and the per-arm setup costs.

func Execute 

func Execute(ctx context.Context, m *Manifest, tasks []Task, prov Provider, grader Grader, opts Options) (*Run, error)

Execute runs the manifest's arms over the corpus and returns the raw ledger. It fails closed: a validation refusal, a provider error, a grader error, or a trial with missing raw evidence aborts the run rather than returning a partially-evidenced report.

func UnmarshalRun 

func UnmarshalRun(b []byte) (*Run, error)

UnmarshalRun parses a run artifact, refusing an unknown schema tag rather than best-effort decoding a shape it does not understand.

type Sampling 

type Sampling struct {
	Temperature float64 `json:"temperature"`
	TopP        float64 `json:"top_p"`
	Seed        int64   `json:"seed"`
}

Sampling pins the decode parameters. Temperature and TopP are pointers-free plain values because 0 is a meaningful setting (temperature 0 is exactly what the Caveman comparator publishes); Seed 0 means "provider default / unseeded" and is recorded as such rather than hidden.

type SelfcheckResult 

type SelfcheckResult struct {
	Schema string  `json:"schema"`
	OK     bool    `json:"ok"`
	Checks []Check `json:"checks"`
	// Report is the human summary of the spine run, captured so the selfcheck
	// artifact is itself the behavioural proof rather than a pass/fail bit that
	// asserts one happened.
	Report        string  `json:"report"`
	SpineIdentity string  `json:"spine_identity"`
	Spine         *Report `json:"spine"`
	// SpineRun carries the raw request/response/judgment evidence behind Spine.
	// The committed selfcheck capture is therefore an inspectable proof, not a
	// report whose measurements have already discarded their source artifacts.
	SpineRun *Run `json:"spine_run"`
}

SelfcheckResult is the whole selfcheck artifact.

func Selfcheck 

func Selfcheck() (*SelfcheckResult, error)

Selfcheck runs the deterministic spine and every fail-closed proof this package owes issue #6676, returning a captured artifact. It never touches the network or the clock, so it is safe on any host and in CI.

The checks, in the order they build on each other:

  1. the spine runs baseline + upstream treatment + fak arms end to end;
  2. changing the model, the prompt, the judge, the corpus, or the capability each moves the manifest identity (the five mutations #6676 names);
  3. a trial with no raw response fails closed;
  4. an arm that bundles two unnamed-apart capabilities fails closed;
  5. resume re-executes nothing and duplicates nothing;
  6. a drifted manifest is refused as incomparable;
  7. the paired order is counterbalanced, not fixed.

type SetupCost 

type SetupCost struct {
	WallMS  float64 `json:"wall_ms"`
	Tokens  int     `json:"tokens"`
	CostUSD float64 `json:"cost_usd"`
	Note    string  `json:"note,omitempty"`
}

SetupCost is an arm's one-time cost — installing a skill, warming a cache, building an index. It is charged to the arm and amortized across its trials in the report, because an arm that saves 5% per turn and costs a minute to set up is not a win at three turns.

type Source 

type Source struct {
	Name                string `json:"name"`
	Repo                string `json:"repo"`
	URL                 string `json:"url,omitempty"`
	SHA                 string `json:"sha"`
	Path                string `json:"path"`
	ContentHash         string `json:"content_hash"`
	License             string `json:"license,omitempty"`
	LicenseBoundary     string `json:"license_boundary,omitempty"`
	LicenseBoundaryHash string `json:"license_boundary_hash,omitempty"`
	LicenseReview       string `json:"license_review,omitempty"`
	RetrievedAt         string `json:"retrieved_at,omitempty"`
	Normalization       string `json:"normalization,omitempty"`
	LocalPath           string `json:"local_path,omitempty"`
}

Source is one pinned upstream input: which repository, at which commit, which path, and the content hash of what was actually retrieved. The content hash is what makes the pin checkable — a repo/SHA pair alone still permits a hand-edited local copy.

type Task 

type Task struct {
	ID     string `json:"id"`
	Input  string `json:"input"`
	Expect string `json:"expect,omitempty"`
}

Task is one corpus item. The runner never interprets Input beyond handing it to the provider — task semantics belong to the corpus importer (#6677) and the judge adapters (#6678), not to the harness that pairs them.

func DemoCorpus 

func DemoCorpus() []Task

DemoCorpus is the pinned three-task corpus the demo manifest declares.

type TokenEvidence 

type TokenEvidence struct {
	Input, Output, CacheWrite, CacheRead int
	ProviderFields                       map[string]any `json:",omitempty"`
}

type TrialResult 

type TrialResult struct {
	ManifestIdentity string   `json:"manifest_identity"`
	ArmID            string   `json:"arm_id"`
	ArmKind          ArmKind  `json:"arm_kind"`
	TaskID           string   `json:"task_id"`
	Trial            int      `json:"trial"`
	Position         int      `json:"position"`
	Response         Response `json:"response"`
	Judgment         Judgment `json:"judgment"`
	// Resumed marks a row carried over from a prior run's ledger rather than
	// re-executed. It is recorded so a resumed report can never be mistaken for
	// a fresh full run.
	Resumed bool `json:"resumed"`
}

TrialResult is one (arm, task, trial) row: the whole evidence chain for a single measurement.

func (TrialResult) Key 

func (t TrialResult) Key() string

Key is the trial's resume identity. Every field in it is part of what makes the row unique; two rows with the same key are the same measurement, which is exactly what resume must not duplicate.

type Trials 

type Trials struct {
	Count       int           `json:"count"`
	Seed        int64         `json:"seed"`
	Order       OrderStrategy `json:"order"`
	Concurrency int           `json:"concurrency"`
}

Trials pins the repetition and pairing plan. Concurrency is recorded, enforced, and identity-bearing: provider throttling and queueing can change measured latency, so a different concurrency is not silently comparable.

type Usage 

type Usage struct {
	InputTokens  int     `json:"input_tokens"`
	OutputTokens int     `json:"output_tokens"`
	CostUSD      float64 `json:"cost_usd"`
}

Usage is the provider-reported accounting. Input and output tokens are kept SEPARATE all the way through the runner and the report: collapsing them into one "tokens saved" number is the single most common way a context-compression claim becomes untrue, since the two have different prices and different causes.

func (Usage) TotalTokens 

func (u Usage) TotalTokens() int

TotalTokens is the labelled sum. It exists so callers never hand-add the two and lose the label.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.