livecodebench

Documentation

Overview

Package livecodebench adapts LiveCodeBench problem suites and run reports into fak-native shapes.

Schema constants:

• SuiteSchema ("fak.livecodebench-suite.v1"): a normalized LCB problem suite (Suite/Problem/TestCase), release-pinned and contest-dated.
• ReportSchema ("fak.livecodebench-report.v1"): a run report (Report/ArmResult/Summary) over a suite.
• FixtureSchema ("fak.livecodebench.fixture.v1") and FixtureReportSchema ("fak.livecodebench.report.v1"): the committed CI smoke-fixture shapes.

Honesty fence: pass-rate/result fields cannot be carried without a named evidence class, and result_claim_allowed cannot be true unless the evidence class is the official lcb_runner grading — a local run can never promote itself into a claimable score.

Index

• Constants
• Variables
• func BuildCodeExecutionPrompt(p Problem, input string, cot bool) (string, error)
• func BuildRepairPrompt(p Problem, wrong, feedback string) (string, error)
• func BuildTestOutputPrompt(p Problem, input string) (string, error)
• func GradeCodeExecution(predicted, expected string) bool
• func GradeTestOutput(predicted, expected string) bool
• func KnownScenario(s Scenario) bool
• func LatestRelease() string
• func MeanPassAtK(tallies []SampleTally, k int) (float64, error)
• func PassAtK(n, c, k int) (float64, error)
• func PendingWork(all []string, done map[string]bool) []string
• func PromotionRequirements() []string
• func RenderOfficialRunContractMarkdown(c OfficialRunContract) string
• func RenderReportMarkdown(r Report) string
• func ScenarioNames(f Fixture) []string
• func ValidateRunReport(r RunReport) error
• func ValidateSmokeReport(r FixtureReport) error
• func WriteCustomEvaluatorInput(w io.Writer, f Fixture) error
• type ArmComparison
  • func CompareArms(raw RawArmReport, fak FakArmReport) ArmComparison
• type ArmResult
• type ArmSummary
• type ArmUsageDelta
• type CacheKeyInput
  • func (in CacheKeyInput) CacheKey() string
• type CacheStats
  • func (s *CacheStats) Hit()
  • func (s CacheStats) HitRate() float64
  • func (s CacheStats) Lookups() int
  • func (s *CacheStats) Miss()
  • func (s CacheStats) Summary() string
• type CodeExecutionCase
• type CodeExecutionProblem
  • func GradeCodeExecutionProblem(p Problem, predicted []string) (CodeExecutionProblem, error)
• type CodeExecutionSummary
  • func BuildCodeExecutionSummary(model string, cot bool, problems []CodeExecutionProblem) (CodeExecutionSummary, error)
• type CodeExtraction
  • func ExtractCode(raw, starter string) CodeExtraction
• type CodegenConfig
• type CodegenGrader
  • func SandboxCodegenGrader(run ExecRunner, opts GradeOptions) CodegenGrader
• type CodegenProblemScore
• type CodegenReport
  • func ScoreCodegen(cfg CodegenConfig, problems []Problem, completions [][]string, ...) (CodegenReport, error)
• type CodegenSummary
• type ContractArm
• type ContractConstants
• type ContractGate
• type ContractGrading
• type ContractProblemSelection
• type ContractUpstreamRef
• type CustomEvalItem
  • func CustomEvaluatorItems(f Fixture) ([]CustomEvalItem, error)
• type DateWindow
  • func NewDateWindow(startDate, endDate, now string) (DateWindow, error)
  • func (w DateWindow) Contains(p Problem) bool
• type DirGenCache
  • func (c DirGenCache) Get(key string) ([]string, bool)
  • func (c DirGenCache) Put(key string, completions []string)
• type ExecRunner
• type FakArmAdjudication
• type FakArmReport
  • func RunFakArm(ctx context.Context, cfg RawArmConfig, release string, problems []Problem, ...) (FakArmReport, error)
• type FakArmSampler
• type FakSampleEvidence
• type Fixture
  • func Load(r io.Reader) (Fixture, error)
  • func LoadFile(path string) (Fixture, error)
  • func (f Fixture) Validate() error
• type FixtureItem
• type FixtureReport
  • func SmokeReport(f Fixture) FixtureReport
• type GenCache
• type GenerationCall
  • func NormalizeProblem(p Problem, scenario Scenario) (GenerationCall, error)
  • func (c GenerationCall) Prompt() string
• type GenerationMessage
• type GradeOptions
• type GradeResult
  • func GradeCode(ctx context.Context, p Problem, code string, run ExecRunner, opts GradeOptions) GradeResult
• type NormalizeOptions
• type OfficialHarness
• type OfficialRunContract
  • func BuildOfficialRunContract(in OfficialRunContractInput) OfficialRunContract
• type OfficialRunContractInput
• type Preflight
  • func BuildPreflight(in PreflightInput) Preflight
• type PreflightGate
• type PreflightInput
• type PreflightProbe
• type Problem
• type ProblemVerdict
  • func ProblemRowsFromSuite(s Suite) []ProblemVerdict
• type Provenance
• type RawArmConfig
• type RawArmProblem
• type RawArmReport
  • func RunRawArm(ctx context.Context, cfg RawArmConfig, problems []Problem, ...) (RawArmReport, error)
• type RawArmSampler
• type RawArmUsage
• type RawCachedResult
  • func RunRawArmCached(ctx context.Context, cfg RawArmConfig, release string, problems []Problem, ...) (RawCachedResult, error)
• type RawSampleUsage
• type ReleaseHeader
  • func PinRelease(selector string, problems int) (ReleaseHeader, error)
• type ReleaseSelection
  • func ResolveRelease(selector string) (ReleaseSelection, error)
• type RepairSource
• type Report
  • func NewReport(s Suite, generatedAt time.Time) Report
  • func (r *Report) MarkOfficiallyGraded() error
  • func (r Report) Validate() error
• type RunConfig
• type RunReport
  • func BuildRunReport(f Fixture, cfg RunConfig) RunReport
• type SampleTally
• type SamplingConfig
  • func DefaultSampling() SamplingConfig
  • func (s SamplingConfig) ArmConfig(model, endpoint string) RawArmConfig
  • func (s *SamplingConfig) RegisterFlags(fs *flag.FlagSet)
  • func (s SamplingConfig) Validate() error
• type Scenario
• type ScenarioReport
• type SelfRepairDelta
  • func BuildSelfRepairDelta(model string, codegenN int, sources []RepairSource) (SelfRepairDelta, error)
• type Suite
  • func LoadSuite(r io.Reader) (Suite, error)
  • func LoadSuiteFile(path string) (Suite, error)
  • func Normalize(ups []UpstreamProblem, opts NormalizeOptions) (Suite, error)
  • func (s Suite) Validate() error
• type Summary
• type TestCase
• type TestOutcome
• type TestOutputCase
• type TestOutputProblem
  • func GradeTestOutputProblem(p Problem, predicted []string) (TestOutputProblem, error)
• type TestOutputSummary
  • func BuildTestOutputSummary(model string, problems []TestOutputProblem) (TestOutputSummary, error)
• type UpstreamProblem
  • func ParseUpstreamRows(data []byte) ([]UpstreamProblem, error)
• type WindowScores
  • func WindowedPassAtK(problems []Problem, tallies []SampleTally, k int, w DateWindow) (WindowScores, error)

Constants

const (
	CodegenDefaultN           = UpstreamDefaultN
	CodegenDefaultTemperature = UpstreamDefaultTemperature
)

Upstream lcb_runner code-generation sampling defaults. A fak codegen run mirrors them so a fak run and a raw `python -m lcb_runner.runner.main` run sample the same way unless explicitly overridden.

const (
	ContractIncomplete = "INCOMPLETE_CONTRACT"
	ContractReady      = "READY_FOR_OFFICIAL_RUN"
)

Closed-vocabulary contract status. It describes whether the run is fully pinned, never whether it passed.

const (
	RawArmName = "raw"
	FakArmName = "fak"
)

FakArmName / RawArmName are the two arm identities a comparison binds.

const (
	ExecAccepted     = "AC"
	ExecWrongAnswer  = "WA"
	ExecTimeLimit    = "TLE"
	ExecRuntimeError = "RE"
	ExecUnavailable  = "UNAVAILABLE"
)

Execution verdicts. Upstream-aligned tokens: AC is the only passing verdict, and TLE is kept distinct from WA so a slow solution is never silently folded into a wrong answer. UNAVAILABLE is the abstention verdict — no sandbox, or no test cases to witness a pass — and always carries Pass=false.

const (
	FixtureSchema       = "fak.livecodebench.fixture.v1"
	FixtureReportSchema = "fak.livecodebench.report.v1"
)

const (
	GenerationRoleSystem = "system"
	GenerationRoleUser   = "user"
)

Roles of the normalized tool-call messages. These are this package's own strings (not the agent client's) so the foundation tier keeps zero upward imports; the CLI maps them 1:1 onto the wire shape it POSTs.

const (
	PreflightBlocked = "BLOCKED_PREFLIGHT"
	PreflightReady   = "READY"
)

Closed-vocabulary preflight status values. They describe whether this host can attempt a LiveCodeBench run, never a benchmark result.

const (
	ReasonUvMissing        = "UV_NOT_INSTALLED"
	ReasonPython311Missing = "PYTHON311_NOT_FOUND"
	ReasonDatasetUnreach   = "HF_DATASET_UNREACHABLE"
	ReasonGatewayUnreach   = "FAK_GATEWAY_UNREACHABLE"
	ReasonSandboxUnavail   = "SANDBOX_UNAVAILABLE"
)

Closed-vocabulary blocking reasons.

const (
	VerdictUngraded = "ungraded"
	VerdictPass     = "pass"
	VerdictFail     = "fail"
)

Per-problem verdict vocabulary, closed. A row is ungraded until the official checker grades its generations; only then may it be pass|fail. The vocabulary describes whether a problem was graded, never whether the run may claim a result — that gate lives on the Report.

const (
	// UpstreamDefaultN mirrors lcb_runner.runner.main's default -n:
	// 10 samples per problem.
	UpstreamDefaultN = 10
	// UpstreamDefaultTemperature mirrors lcb_runner's default --temperature.
	UpstreamDefaultTemperature = 0.2
	// DefaultSamplingConcurrency bounds in-flight gateway requests (the
	// closed-API --multiprocess analog) so a run respects gateway rate limits.
	DefaultSamplingConcurrency = 8
	// DefaultMaxSampleRetries is the per-sample retry budget: a failed sample
	// is retried up to this many times, each retry counted in
	// RawArmUsage.Retries; exhausting the budget aborts the run loudly. A
	// failed sample is never silently dropped.
	DefaultMaxSampleRetries = 2
)

const (
	SuiteSchema  = "fak.livecodebench-suite.v1"
	ReportSchema = "fak.livecodebench-report.v1"

	Benchmark = "livecodebench"
)

const (
	EvidenceFixtureSmoke      = "fixture-smoke"
	EvidenceLocalUngraded     = "local-ungraded"
	EvidenceOfficialLCBRunner = "official-lcb-runner-graded"
)

Evidence classes, weakest to strongest. Only the official lcb_runner grading can back a claimable result.

const ABComparisonSchema = "fak.livecodebench-ab-comparison.v1"

ABComparisonSchema identifies the raw-vs-fak arm comparison artifact.

const CodeExecutionSummarySchema = "fak.livecodebench.codeexecution.v1"

CodeExecutionSummarySchema tags a code-execution summary artifact.

const DefaultGradeTimeout = 6 * time.Second

DefaultGradeTimeout mirrors lcb_runner's per-test wall-clock default (6s).

const EngineInKernel = "inkernel"

EngineInKernel is the engine id of fak's own decode — the pure-kernel serving path (`fak serve --gguf --engine inkernel`), with no external serving engine (SGLang / vLLM / a hosted proxy) anywhere in the request path (#2107).

const GenerationLanguage = "python"

GenerationLanguage is the language every normalized call targets. Upstream code_generation_lite is Python-only, so the language hint is threaded as a constant into the code fences rather than carried per problem.

const OfficialRunContractSchema = "fak.livecodebench-official-run-contract.v1"

OfficialRunContractSchema identifies the LiveCodeBench official-run contract: a machine-readable statement of the two-arm run shape (raw lcb_runner vs fak-native), the constants pinned before generating anything, the exact official grading handoff, and the gates that must be green before any pass-rate is claimable. It performs no run and carries no score — its whole job is to make the run reproducible and to keep result_claim_allowed=false until the official evaluator has graded the exact saved generations. Part of the LiveCodeBench epic #2085 (#2110), gating the DGX GLM-5.2 run #3060.

const PassRateDeltaUngraded = "ungraded: no pass-rate delta may be claimed until the official lcb_runner evaluator grades both arms' saved generations"

PassRateDeltaUngraded is the only pass-rate delta this package will ever state on its own authority: none. Only the official lcb_runner evaluator, grading the exact saved generations of BOTH arms, may back a claim that adjudication changed the pass rate (#2105 acceptance).

const PendingGPURun = "pending GPU run"

PendingGPURun is the only value a pure-kernel pass rate may carry until a real GPU run through the in-kernel path returns a number from the official evaluator. The device kernels are argmax-exact and `fak serve --engine inkernel` is a landed serving path, but neither fact is an LCB codegen pass rate (#2107).

const PreflightSchema = "fak.livecodebench-preflight.v1"

PreflightSchema identifies the host-readiness preflight artifact that gates a LiveCodeBench run (#2111, part of the LiveCodeBench epic #2085).

const ReleaseLatest = "release_latest"

ReleaseLatest is upstream's alias for the newest release. It is always resolved to a concrete release and recorded explicitly, never left implicit.

const RunReportSchema = "fak.livecodebench.run.v1"

RunReportSchema tags the artifact a fixture run writes.

const SelfRepairDeltaSchema = "fak.livecodebench.selfrepair.v1"

SelfRepairDeltaSchema tags a self-repair delta artifact.

const SelfRepairRepairN = 1

SelfRepairRepairN is the fixed fan-out of the repair pass. Upstream LCB self-repair repairs each source codegen generation exactly ONCE — the scenario is defined at n=1, and a wider repair fan-out is a different experiment, not a self-repair run. It is recorded on every delta so a reader can see the repair pass ran at n=1.

const TestOutputSummarySchema = "fak.livecodebench.testoutput.v1"

TestOutputSummarySchema tags a test-output-prediction summary artifact.

Variables

var RequiredScenarios = []string{
	"codegeneration",
	"selfrepair",
	"testoutputprediction",
	"codeexecution",
}

Functions

func BuildCodeExecutionPrompt

func BuildCodeExecutionPrompt(p Problem, input string, cot bool) (string, error)

BuildCodeExecutionPrompt renders the code-execution request for ONE input: the program and the specific input, asking the model for the exact output of executing the program on that input. When cot is true the request asks the model to reason step by step before answering (the upstream chain-of-thought mode); when false it asks for the output directly. This is the upstream code-execution prompt shape (program + input -> output) rendered purely so it is unit-tested without a model call. It refuses an empty program or an empty input — there is nothing to execute, or nothing to execute it on, otherwise.

func BuildRepairPrompt

func BuildRepairPrompt(p Problem, wrong, feedback string) (string, error)

BuildRepairPrompt renders the self-repair request for one failing generation: the original problem, the prior WRONG solution, and the failing-test feedback, asking the model for a corrected solution. This is the upstream self-repair prompt shape (problem + wrong attempt + observed failure) rendered purely so it is unit-tested without a model call. It refuses an empty prior generation or empty feedback — a repair with nothing to repair from, or no observed failure, is not a self-repair request.

func BuildTestOutputPrompt

func BuildTestOutputPrompt(p Problem, input string) (string, error)

BuildTestOutputPrompt renders the test-output-prediction request for ONE input: the problem, its code (starter_code, when the problem carries it), and the specific input, asking the model for the exact program output and nothing else. This is the upstream output-prediction prompt shape (problem + code + input -> output) rendered purely so it is unit-tested without a model call. It refuses an empty problem prompt or an empty input — there is nothing to predict from, or nothing to predict for, otherwise.

func GradeCodeExecution

func GradeCodeExecution(predicted, expected string) bool

GradeCodeExecution compares one predicted output against the expected output under normalizeTestOutput (line-ending- and trailing-whitespace-insensitive), returning true on an exact normalized match.

func GradeTestOutput

func GradeTestOutput(predicted, expected string) bool

GradeTestOutput compares one predicted output against the expected output under normalizeTestOutput (line-ending- and trailing-whitespace-insensitive), returning true on an exact normalized match.

func KnownScenario

func KnownScenario(s Scenario) bool

func LatestRelease

func LatestRelease() string

LatestRelease returns the concrete release ReleaseLatest resolves to.

func MeanPassAtK

func MeanPassAtK(tallies []SampleTally, k int) (float64, error)

MeanPassAtK is the benchmark-level score: the mean of PassAtK over each problem's (Samples, Correct) tally, matching how LiveCodeBench aggregates a scenario's pass@k. It errors if the tally set is empty or any tally is invalid.

func PassAtK

func PassAtK(n, c, k int) (float64, error)

PassAtK computes the unbiased pass@k estimator of Chen et al. (2021, "Evaluating Large Language Models Trained on Code"), the same estimator LiveCodeBench's lcb_runner uses to score generations: given n samples of which c passed, it returns the probability that a uniformly random size-k subset of the samples contains at least one passing sample, in expectation over subsets:

pass@k = 1 - C(n-c, k) / C(n, k)

It is evaluated in the numerically stable product form 1 - prod_{i=n-c+1}^{n} (1 - k/i) so it does not overflow on large n. When n-c < k every size-k subset must include a passing sample, so the value is 1.

Tolerance vs upstream: this is an exact evaluation of the estimator, so the only divergence from lcb_runner is which samples are marked correct. LCB notes a timing-induced <0.5pt variance in that upstream correctness labeling (a sandbox may time out a borderline-correct sample); the estimator itself carries no additional error.

func PendingWork

func PendingWork(all []string, done map[string]bool) []string

PendingWork is the `--continue-existing` resume filter: given the full ordered list of problem ids a run must cover and the set already completed by a prior (possibly interrupted) run, it returns the ids still to do, in the original order and with no duplicates. An interrupted run resumed with the same work list therefore never re-generates a problem it already finished. Duplicate ids in `all` collapse to their first occurrence, so a malformed work list cannot cause the same problem to be scored twice.

func PromotionRequirements

func PromotionRequirements() []string

PromotionRequirements is the canonical checklist a local LiveCodeBench number must clear before it may be promoted to a claimable score: the pinned problem ids, the release + contest-date window, both arms' saved generations, the official grader's output, and the same-config identity across arms. Every report stamps it verbatim; it returns a fresh slice each call so a caller cannot mutate the shared contract.

func RenderOfficialRunContractMarkdown

func RenderOfficialRunContractMarkdown(c OfficialRunContract) string

RenderOfficialRunContractMarkdown renders the contract as human-readable markdown, mirroring the terminalbench contract layout.

func RenderReportMarkdown

func RenderReportMarkdown(r Report) string

RenderReportMarkdown renders a Report as human-readable markdown: an evidence-class / claim-boundary banner, a summary table, per-scenario counts, the raw-vs-fak arm deltas, and the per-problem verdict rows that link each question_id to its verdict and evidence id. It is a pure function of the report — no clock, no I/O — so its output is golden-file testable.

func ScenarioNames

func ScenarioNames(f Fixture) []string

func ValidateRunReport

func ValidateRunReport(r RunReport) error

ValidateRunReport enforces the honesty fence on a run report: a fixture run can never promote itself into a claimable score, and it must have scored at least one question of the requested scenario.

func ValidateSmokeReport

func ValidateSmokeReport(r FixtureReport) error

func WriteCustomEvaluatorInput

func WriteCustomEvaluatorInput(w io.Writer, f Fixture) error

WriteCustomEvaluatorInput writes the custom_evaluator input JSON for a fixture to w, ordered to match the benchmark problems. Grade the emitted file with the official checker:

python -m lcb_runner.runner.custom_evaluator \
    --custom_output_file <input.json> \
    --release_version <release_vN>

The result is only claimable once that official run produces its grading — a local export can never promote itself into a score.

Types

type ArmComparison

type ArmComparison struct {
	Schema             string             `json:"schema"`
	Raw                ArmSummary         `json:"raw"`
	Fak                ArmSummary         `json:"fak"`
	SameProblemIDs     bool               `json:"same_problem_ids"`
	SamePromptHash     bool               `json:"same_prompt_hash"`
	SameModel          bool               `json:"same_model"`
	SameN              bool               `json:"same_n"`
	SameTemperature    bool               `json:"same_temperature"`
	SameRelease        bool               `json:"same_release"`
	Mismatches         []string           `json:"mismatches,omitempty"`
	UsageDelta         ArmUsageDelta      `json:"usage_delta"` // fak minus raw
	FakAdjudication    FakArmAdjudication `json:"fak_adjudication"`
	PassRateDelta      string             `json:"pass_rate_delta"`      // always PassRateDeltaUngraded here
	ResultClaimAllowed bool               `json:"result_claim_allowed"` // always false here
	ClaimBoundary      string             `json:"claim_boundary"`
}

ArmComparison is the two-arm A/B artifact (#2105): per-arm summaries, the cross-arm identity assertions the acceptance demands (SameProblemIDs / SamePromptHash, plus model / n / temperature / release), the usage delta, and the fak arm's adjudication evidence. ResultClaimAllowed is always false and PassRateDelta is always the ungraded sentinel — grading authority stays with the official evaluator.

func CompareArms

func CompareArms(raw RawArmReport, fak FakArmReport) ArmComparison

CompareArms builds the raw-vs-fak comparison from the two arm artifacts alone: every assertion is checked against what the reports actually recorded (never against shared inputs), so a stale or foreign report cannot pass. It is pure and never claims a pass-rate delta.

type ArmResult

type ArmResult struct {
	Arm         string   `json:"arm"`
	Scenario    Scenario `json:"scenario"`
	Problems    int      `json:"problems"`
	Generations int      `json:"generations"`
	Graded      int      `json:"graded"`
	Pass1       float64  `json:"pass_1,omitempty"`
	Pass5       float64  `json:"pass_5,omitempty"`
}

ArmResult is one (arm, scenario) result cell: raw vs fak, per scenario.

type ArmSummary

type ArmSummary struct {
	Arm         string      `json:"arm"`
	Model       string      `json:"model"`
	Endpoint    string      `json:"endpoint"`
	N           int         `json:"n"`
	Temperature float64     `json:"temperature"`
	Release     string      `json:"release,omitempty"`
	Problems    int         `json:"problems"`
	Usage       RawArmUsage `json:"usage"`
}

ArmSummary is the per-arm run identity + usage the comparison emits for each arm.

type ArmUsageDelta

type ArmUsageDelta struct {
	Samples            int `json:"samples"`
	PromptTokens       int `json:"prompt_tokens"`
	CompletionTokens   int `json:"completion_tokens"`
	CachedPromptTokens int `json:"cached_prompt_tokens"`
}

ArmUsageDelta is fak-minus-raw token accounting: the observable cost of the adjudicated path, stated without any quality claim.

type CacheKeyInput

type CacheKeyInput struct {
	Model       string  // model identity the completion was generated by
	Prompt      string  // the fully-rendered prompt sent to the model
	N           int     // samples requested per problem
	Temperature float64 // sampling temperature
	Release     string  // LCB dataset release the problem is pinned to
}

CacheKeyInput is the identity of one generation request: the exact inputs that, if unchanged, make a prior completion safe to reuse instead of re-spending tokens. It mirrors the lcb_runner `--use_cache` contract (#2108): the cache key includes model + prompt + n + temperature + release, so a re-run with the same knobs reuses the cached completion and a run that changes ANY of them misses and regenerates. Nothing else is part of the identity — two runs that differ only in where the artifact is written share a key.

func (CacheKeyInput) CacheKey

func (in CacheKeyInput) CacheKey() string

CacheKey is the deterministic content address of a generation request. It is a SHA-256 over a canonical, length-prefixed encoding of every identity field, so the boundary between fields cannot be forged: a model "a" + prompt "bc" can never collide with a model "ab" + prompt "c". The temperature is encoded with full float precision (strconv 'g', -1) so 0.2 and 0.20000001 are distinct keys. Same inputs -> same key on every host and every run; that stability is what lets a re-run recognize a cached completion.

type CacheStats

type CacheStats struct {
	Hits   int `json:"hits"`
	Misses int `json:"misses"`
}

CacheStats is honest cache-hit accounting (#2108): it counts only genuine cache lookups, so the surfaced hit rate can never be a vanity number. A "hit" is a completion actually reused; a "miss" is a lookup that had to regenerate. Work that was never looked up (e.g. a problem outside the run) is not counted at all — it can neither inflate nor deflate the rate.

func (*CacheStats) Hit

func (s *CacheStats) Hit()

Hit records a reused completion.

func (CacheStats) HitRate

func (s CacheStats) HitRate() float64

HitRate is Hits/Lookups in [0,1]. With zero lookups it is 0, never an undefined-as-100% flourish: a run that reused nothing reports a 0% hit rate, not a vacuous perfect one.

func (CacheStats) Lookups

func (s CacheStats) Lookups() int

Lookups is the denominator the rate is honest against: only requests actually checked against the cache.

func (*CacheStats) Miss

func (s *CacheStats) Miss()

Miss records a lookup that had to regenerate.

func (CacheStats) Summary

func (s CacheStats) Summary() string

Summary renders the accounting as absolute counts first, with the rate stated against the lookup count so the reader can always see the denominator behind it.

type CodeExecutionCase

type CodeExecutionCase struct {
	Input     string `json:"input"`
	Expected  string `json:"expected"`
	Predicted string `json:"predicted"`
	Correct   bool   `json:"correct"`
}

CodeExecutionCase is the graded result for one problem input: the input, its expected output, the model's predicted output, and whether they matched under GradeCodeExecution.

type CodeExecutionProblem

type CodeExecutionProblem struct {
	QuestionID string              `json:"question_id"`
	Cases      []CodeExecutionCase `json:"cases"`
	Correct    bool                `json:"correct"`
}

CodeExecutionProblem is the per-problem execution result: every input's predicted vs expected output and whether the model got ALL of them right. A problem is Correct only when every one of its inputs is predicted correctly — a partially-right multi-input problem is a miss, matching upstream all-or-nothing scoring.

func GradeCodeExecutionProblem

func GradeCodeExecutionProblem(p Problem, predicted []string) (CodeExecutionProblem, error)

GradeCodeExecutionProblem grades one problem's predictions: predicted[i] is the model's output for the problem's i-th public test-case input, graded against that case's expected Output. It refuses when the problem carries no test cases (nothing to execute) or when the prediction count does not match the input count (a prediction is required for every input — a short slice would silently score missing inputs as passes). The returned problem is Correct only when every input matched.

type CodeExecutionSummary

type CodeExecutionSummary struct {
	Schema             string                 `json:"schema"`
	Scenario           Scenario               `json:"scenario"`
	Model              string                 `json:"model,omitempty"`
	CoT                bool                   `json:"cot"`
	Problems           []CodeExecutionProblem `json:"problems"`
	Correct            int                    `json:"correct"`
	Accuracy           float64                `json:"accuracy"`
	EvidenceClass      string                 `json:"evidence_class"`
	ResultClaimAllowed bool                   `json:"result_claim_allowed"`
}

CodeExecutionSummary is the machine-readable result of a code-execution run: the per-problem correctness rows and the summary accuracy (fraction of problems all of whose inputs were predicted correctly), tagged with the CoT mode the run used so a CoT and a non-CoT run are recorded separately, plus the evidence gate.

func BuildCodeExecutionSummary

func BuildCodeExecutionSummary(model string, cot bool, problems []CodeExecutionProblem) (CodeExecutionSummary, error)

BuildCodeExecutionSummary aggregates graded per-problem executions into a run summary: the accuracy is the fraction of problems all of whose inputs the model predicted correctly. The cot flag is recorded verbatim so a CoT run and a non-CoT run over the same problems produce two separately-tagged summaries. It refuses an empty set (no problems graded) or a problem with a blank question_id. Evidence is gated — ResultClaimAllowed stays false until the official lcb_runner grades the same predictions.

type CodeExtraction

type CodeExtraction struct {
	// Code is the extracted program, ready to hand to the grader. Internal
	// whitespace and indentation are preserved verbatim (code is significant); it
	// is empty when NoCode is true.
	Code string
	// Language is the fenced language tag, lower-cased ("python", "cpp", ...), or
	// "" when the answer was fenced without a tag. Meaningless when NoCode is true.
	Language string
	// NoCode is the explicit no-gradeable-code verdict.
	NoCode bool
}

CodeExtraction is the result of pulling the gradeable code out of ONE raw model generation. NoCode is an explicit verdict — the output carried nothing to grade (empty, prose-only, or an empty fenced block) — kept distinct from an extracted empty string so the grader can score "no code" as a miss instead of crashing on it or silently treating it as a pass.

func ExtractCode

func ExtractCode(raw, starter string) CodeExtraction

ExtractCode pulls the final gradeable code block out of a raw model generation, mirroring the lcb_runner fence convention: a model reasons in prose and emits its answer as a fenced code block, so the LAST closed ``` fence wins over any earlier ones (a self-repair or worked-example block never shadows the final answer). Behaviour by shape:

• fenced with a language tag (```python\n...\n```): the tag is captured into Language and stripped from Code.
• fenced without a tag (```\n...\n```): Code is the block, Language is "".
• multi-block: only the last closed fence pair is returned.
• starter-merge: when starter (the problem's starter_code) is non-empty and its signature is absent from the extracted block, starter is prepended so the grader sees the complete program; when the block already carries the signature it is left untouched (no duplication).
• empty / prose-only / an empty fenced block: NoCode, so the grader records an explicit miss rather than crashing or scoring garbage as a pass.

Unfenced prose is deliberately NOT salvaged as code: under the LCB convention a model that does not fence its answer has not produced a gradeable one, and guessing would fabricate a pass. starter is the problem's starter_code ("" for a pure code-generation problem).

type CodegenConfig

type CodegenConfig struct {
	Scenario    Scenario
	N           int
	Temperature float64
}

CodegenConfig pins the codegen scenario's recorded sampling parameters. N and Temperature are recorded verbatim in the report so a reader can trace which sampling regime produced the pass rates; Scenario is carried for the report header (it is always ScenarioCodeGeneration for this scorer).

type CodegenGrader

type CodegenGrader func(p Problem, code string) (pass bool, err error)

CodegenGrader grades ONE extracted candidate program for a problem, returning whether it passes the problem's tests. It is injected (like RawArmSampler) so the scenario is unit-tested without executing untrusted code and the CLI can plug the official lcb_runner checker or a sandbox. A NoCode extraction is never handed to the grader — it is scored an automatic miss.

func SandboxCodegenGrader

func SandboxCodegenGrader(run ExecRunner, opts GradeOptions) CodegenGrader

SandboxCodegenGrader adapts the sandbox execution grader to the injected CodegenGrader seam ScoreCodegen consumes: a sample passes only on an AC verdict from a live sandbox. When run is nil the sandbox is unavailable, so the adapter returns an error rather than a fabricated miss — the run aborts instead of silently scoring every sample zero, keeping the honesty fence intact from the grader through the scorer.

type CodegenProblemScore

type CodegenProblemScore struct {
	QuestionID string  `json:"question_id"`
	Samples    int     `json:"samples"`   // n: completions graded for this problem
	Extracted  int     `json:"extracted"` // samples that yielded gradeable code
	Correct    int     `json:"correct"`   // c: samples that passed grading
	Pass1      float64 `json:"pass_1"`
	Pass5      float64 `json:"pass_5"`
}

CodegenProblemScore is one problem's per-problem codegen result: n samples of which Correct passed grading (Extracted of them yielded gradeable code at all), and the pass@1 / pass@5 that (n, Correct) implies.

type CodegenReport

type CodegenReport struct {
	Scenario    Scenario              `json:"scenario"`
	N           int                   `json:"n"`
	Temperature float64               `json:"temperature"`
	Problems    []CodegenProblemScore `json:"problems"`
	Summary     CodegenSummary        `json:"summary"`
}

CodegenReport is the codegen scenario's end-to-end result: the recorded sampling config, one per-problem row, and the summary pass@1 / pass@5.

func ScoreCodegen

func ScoreCodegen(cfg CodegenConfig, problems []Problem, completions [][]string, grade CodegenGrader) (CodegenReport, error)

ScoreCodegen runs the code-generation scenario end to end over the completions sampled for each problem. completions[i] holds the samples generated for problems[i]; the two slices must be the same length and in the same order. For each sample it extracts the gradeable code (ExtractCode with the problem's starter_code), scores a NoCode extraction an automatic miss, and otherwise hands the extracted program to the injected grader; the passing count c and the sample count n feed the unbiased PassAtK estimator for per-problem pass@1 / pass@5, and the summary is their mean over problems (MeanPassAtK). cfg.N and cfg.Temperature are recorded verbatim.

Every problem must carry at least five samples so pass@5 is well-defined (upstream samples n=10); the grader is required and its first error aborts the run with problem/sample context.

type CodegenSummary

type CodegenSummary struct {
	Problems    int     `json:"problems"`
	Generations int     `json:"generations"`
	Graded      int     `json:"graded"`
	Pass1       float64 `json:"pass_1"`
	Pass5       float64 `json:"pass_5"`
}

CodegenSummary is the benchmark-level fold: the mean of the per-problem pass@1 / pass@5 across every scored problem, matching how lcb_runner aggregates a scenario.

type ContractArm

type ContractArm struct {
	Name              string   `json:"name"`
	Harness           string   `json:"harness"`
	Commands          []string `json:"commands"`
	OutputDir         string   `json:"output_dir"`
	RequiredArtifacts []string `json:"required_artifacts"`
	Notes             string   `json:"notes,omitempty"`
}

ContractArm is one generation arm: the raw official lcb_runner arm or the fak-native arm. Neither arm's command grades — grading is the single official authority in ContractGrading.

type ContractConstants

type ContractConstants struct {
	ReleaseSelector string `json:"release_selector"`
	ReleaseVersion  string `json:"release_version"`
	Scenario        string `json:"scenario"`
	StartDate       string `json:"start_date"`
	EndDate         string `json:"end_date"`
	Model           string `json:"model"`
	Engine          string `json:"engine,omitempty"`
	ServingBackend  string `json:"serving_backend,omitempty"`
	Gateway         string `json:"gateway,omitempty"`
	RunDir          string `json:"run_dir"`
}

ContractConstants are the run identities that must be pinned before any generation and kept identical across both arms (#3060 "constants pinned").

type ContractGate

type ContractGate struct {
	Name   string `json:"name"`
	OK     bool   `json:"ok"`
	Detail string `json:"detail,omitempty"`
}

ContractGate is one readiness condition. All gates OK => READY_FOR_OFFICIAL_RUN.

type ContractGrading

type ContractGrading struct {
	Authority              string   `json:"authority"`
	RawEvaluateCommand     string   `json:"raw_evaluate_command"`
	CustomEvaluatorCommand string   `json:"custom_evaluator_command"`
	ComputeScoresCommand   string   `json:"compute_scores_command"`
	RequiredArtifacts      []string `json:"required_artifacts"`
	Detail                 string   `json:"detail"`
}

ContractGrading is the sole result-bearing authority: the official LiveCodeBench evaluator over the exact saved generations from each arm.

type ContractProblemSelection

type ContractProblemSelection struct {
	CandidateSuite         string   `json:"candidate_suite,omitempty"`
	CandidateProblemIDs    []string `json:"candidate_problem_ids,omitempty"`
	SameProblemIDsRequired bool     `json:"same_problem_ids_required"`
	SamePromptHashRequired bool     `json:"same_prompt_hash_required"`
	SameReleaseRequired    bool     `json:"same_release_required"`
	SameModelRequired      bool     `json:"same_model_required"`
	SameScenarioRequired   bool     `json:"same_scenario_required"`
}

ContractProblemSelection fixes which problems both arms score and the cross-arm identity requirements (#3060 SameProblemIDs / SamePromptHash).

type ContractUpstreamRef

type ContractUpstreamRef struct {
	Name  string `json:"name"`
	URL   string `json:"url"`
	Notes string `json:"notes,omitempty"`
}

ContractUpstreamRef is a pin to the upstream harness command a reader can verify the contract against.

type CustomEvalItem

type CustomEvalItem struct {
	QuestionID string   `json:"question_id"`
	CodeList   []string `json:"code_list"`
}

CustomEvalItem is one entry of the input the upstream lcb_runner.runner.custom_evaluator consumes: a question_id paired with the model's code generations for that problem. The JSON shape is exactly [{"question_id": ..., "code_list": [...]}], the format the OFFICIAL checker grades — this is the bridge that turns local generations into a real, claimable result.

func CustomEvaluatorItems

func CustomEvaluatorItems(f Fixture) ([]CustomEvalItem, error)

CustomEvaluatorItems projects a fixture's items into custom_evaluator input, preserving fixture order so the emitted slice lines up with the benchmark problems, and passing each question_id through unchanged. An item with no code_list is rejected: the custom evaluator has nothing to grade for it, and silently emitting an empty code_list would fabricate an ungradeable entry.

type DateWindow

type DateWindow struct {
	Start time.Time // inclusive lower bound; the zero time means "open" (no lower bound)
	End   time.Time // inclusive upper bound
}

DateWindow is a contamination-free contest-date window [Start, End]. Scoring a run over only the problems whose contest_date falls in the window lets a result state a window that post-dates a model's training cut, instead of claiming credit for problems the model may have memorized — the LiveCodeBench compute_scores.py contamination control.

End defaults to a caller-supplied "now" when the window leaves it open (the upstream convention: the window runs up to the present). That now is INJECTED, never read from the wall clock, so a windowed score is byte-for-byte reproducible in a test or a re-run — the determinism requirement fak carries everywhere.

func NewDateWindow

func NewDateWindow(startDate, endDate, now string) (DateWindow, error)

NewDateWindow parses YYYY-MM-DD start/end/now into a window. An empty start is an open lower bound (time zero). An empty end defaults to now, so now is REQUIRED (and must parse) whenever end is empty. It errors on an unparseable date or a window whose end precedes its start.

func (DateWindow) Contains

func (w DateWindow) Contains(p Problem) bool

Contains reports whether a problem's contest_date falls within [Start, End] inclusive. A problem with no contest_date — or one that does not parse — is in NO window: an undated problem cannot be certified contamination-free, so it is excluded from the windowed set (and only the windowed set; it still counts in the full-set rate). The Start zero value is an open lower bound.

type DirGenCache

type DirGenCache struct {
	Dir string
}

DirGenCache is a GenCache holding one JSON file per key under Dir. Keys are CacheKeyInput.CacheKey() values ("lcbgen_" + hex), so they are always filesystem-safe. An unreadable or corrupt entry reads as a miss, never an error: the cache can only save tokens, not fail a run.

func (DirGenCache) Get

func (c DirGenCache) Get(key string) ([]string, bool)

Get returns the completions stored for key, or ok=false on any failure.

func (DirGenCache) Put

func (c DirGenCache) Put(key string, completions []string)

Put persists the completions for key. Failures are swallowed by design: generation already succeeded, so the worst outcome is a miss next run.

type ExecRunner

type ExecRunner func(ctx context.Context, code, stdin string, timeout time.Duration) (stdout string, timedOut bool, err error)

ExecRunner executes candidate code against one test's stdin under the given per-test timeout and returns the program's stdout. timedOut reports that the program exceeded the timeout (graded TLE); a non-nil err that is not a timeout is a runtime error (graded RE). It is injected: the CLI plugs a Docker/POSIX sandbox, tests plug a pure oracle.

type FakArmAdjudication

type FakArmAdjudication struct {
	AdjudicatedSamples int `json:"adjudicated_samples"`
	Adjudications      int `json:"adjudications"`
	Denied             int `json:"denied"`
	SafeResolves       int `json:"safe_resolves"`
	ResultAdmissions   int `json:"result_admissions"`
}

FakArmAdjudication is the run's adjudication evidence folded across every sample. It is evidence of the adjudicated path having been exercised — never a claim that adjudication changed any pass rate.

type FakArmReport

type FakArmReport struct {
	Arm          string             `json:"arm"` // always "fak"
	Model        string             `json:"model"`
	Endpoint     string             `json:"endpoint"`
	N            int                `json:"n"`
	Temperature  float64            `json:"temperature"`
	Seed         int64              `json:"seed,omitempty"` // 0 = provider default, omitted
	Concurrency  int                `json:"concurrency"`
	MaxRetries   int                `json:"max_retries"` // per-sample retry budget the run honored (#2106)
	Release      string             `json:"release,omitempty"`
	Problems     []RawArmProblem    `json:"problems"`
	Usage        RawArmUsage        `json:"usage"`
	Adjudication FakArmAdjudication `json:"adjudication"`
}

FakArmReport is the machine-readable result of the fak arm. Its run-identity fields mirror RawArmReport exactly so CompareArms can assert the two arms ran the same problems / model / n / temperature / release.

func RunFakArm

func RunFakArm(ctx context.Context, cfg RawArmConfig, release string, problems []Problem, sample FakArmSampler) (FakArmReport, error)

RunFakArm fans the sampler out over every (problem, sample) pair with the SAME fan-out semantics as RunRawArm (bounded concurrency, first error aborts, deterministic problem/sample ordering), folding the per-sample adjudication evidence into the report. The fan-out is delegated to RunRawArm so the two arms cannot drift.

type FakArmSampler

type FakArmSampler func(ctx context.Context, p Problem, i int) (content string, u RawSampleUsage, ev FakSampleEvidence, err error)

FakArmSampler produces ONE completion for problem p at sample index i via the adjudicated gateway path, returning the sampler-normalized usage plus the adjudication evidence the response carried. Tests inject a deterministic stub.

type FakSampleEvidence

type FakSampleEvidence struct {
	Adjudicated      bool // the response carried the fak adjudication extension
	Adjudications    int  // tool-call adjudications recorded on the response
	Denied           int  // adjudications with admitted=false (policy denials)
	SafeResolves     int  // adjudications carrying repaired arguments (safe-resolve)
	ResultAdmissions int  // inbound result admissions recorded (evidence trail)
}

FakSampleEvidence is the adjudication evidence for ONE sample, extracted by the sampler from the gateway's `fak` response extension. A zero value means the response carried no fak extension (Adjudicated=false): the request was served, but nothing on this sample witnessed the adjudicated path.

type Fixture

type Fixture struct {
	Schema         string        `json:"schema"`
	ReleaseVersion string        `json:"release_version"`
	StartDate      string        `json:"start_date"`
	EndDate        string        `json:"end_date"`
	Items          []FixtureItem `json:"items"`
}

func Load

func Load(r io.Reader) (Fixture, error)

func LoadFile

func LoadFile(path string) (Fixture, error)

func (Fixture) Validate

func (f Fixture) Validate() error

type FixtureItem

type FixtureItem struct {
	QuestionID string   `json:"question_id"`
	Scenario   string   `json:"scenario"`
	Prompt     string   `json:"prompt"`
	CodeList   []string `json:"code_list,omitempty"`
}

type FixtureReport

type FixtureReport struct {
	Schema             string           `json:"schema"`
	FixtureSchema      string           `json:"fixture_schema"`
	ReleaseVersion     string           `json:"release_version"`
	StartDate          string           `json:"start_date"`
	EndDate            string           `json:"end_date"`
	Questions          int              `json:"questions"`
	Scenarios          []ScenarioReport `json:"scenarios"`
	ResultClaimAllowed bool             `json:"result_claim_allowed"`
	EvidenceClass      string           `json:"evidence_class"`
	PromotionRequired  []string         `json:"promotion_required"`
}

func SmokeReport

func SmokeReport(f Fixture) FixtureReport

type GenCache

type GenCache interface {
	Get(key string) ([]string, bool)
	Put(key string, completions []string)
}

GenCache stores the n completions generated for one cache key (the CacheKeyInput identity: model + prompt + n + temperature + release). A Get that returns ok=false means the caller must regenerate; Put is best-effort — a failed persist costs a future miss, never this run's correctness.

type GenerationCall

type GenerationCall struct {
	Scenario     Scenario            `json:"scenario"`
	QuestionID   string              `json:"question_id"`
	Messages     []GenerationMessage `json:"messages"`
	PromptSHA256 string              `json:"prompt_sha256"`
}

GenerationCall is the normalized prompt + tool-call for one (problem, scenario): the messages the gateway request carries, and the stable identity of the exact rendered prompt. Two calls are the same workload if and only if their PromptSHA256 match — this is the input the A/B arms must agree on for SamePromptHash to hold.

func NormalizeProblem

func NormalizeProblem(p Problem, scenario Scenario) (GenerationCall, error)

NormalizeProblem turns one suite Problem into the generation tool-call fak sends through the gateway for the given scenario: the per-scenario system + user messages, and the SHA-256 identity of the exact rendered prompt. It refuses an unknown scenario, and refuses a problem whose own Scenario tag disagrees with the requested one — silently normalizing a problem under the wrong scenario would mint a plausible-but-wrong workload identity.

func (GenerationCall) Prompt

func (c GenerationCall) Prompt() string

Prompt is the canonical rendering the hash covers: every message as role + NUL + content, messages joined by double-NUL. NUL framing cannot collide with message text, so moving bytes between messages (or between role and content) always changes the identity.

type GenerationMessage

type GenerationMessage struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

GenerationMessage is one chat message of the normalized tool-call.

type GradeOptions

type GradeOptions struct {
	// Timeout is the per-test wall-clock limit (upstream --timeout). Zero falls
	// back to DefaultGradeTimeout so a caller cannot accidentally run unbounded.
	Timeout time.Duration
	// NumProcessEvaluate mirrors upstream --num_process_evaluate: the maximum
	// number of test cases evaluated concurrently. Values <=1 grade serially.
	// Aggregation is index-ordered regardless, so the verdict is deterministic.
	NumProcessEvaluate int
}

GradeOptions carries the lcb_runner-parity execution knobs.

type GradeResult

type GradeResult struct {
	QuestionID string        `json:"question_id"`
	Verdict    string        `json:"verdict"`
	Pass       bool          `json:"pass"`
	TestsRun   int           `json:"tests_run"`
	TestsTotal int           `json:"tests_total"`
	Outcomes   []TestOutcome `json:"outcomes,omitempty"`
	// OfficialHarnessAvailable is false when no sandbox runner was supplied; the
	// grader then abstains (Verdict=UNAVAILABLE, Pass=false). It is the
	// machine-readable honesty fence a report carries — an unavailable sandbox
	// degrades here, it never fabricates a pass.
	OfficialHarnessAvailable bool `json:"official_harness_available"`
}

GradeResult is a candidate program's execution grade for one problem.

func GradeCode

func GradeCode(ctx context.Context, p Problem, code string, run ExecRunner, opts GradeOptions) GradeResult

GradeCode runs code against p's public + private test cases through the injected sandbox runner and folds the per-test outcomes into one verdict: AC only when every test matches within the time limit, otherwise the first non-AC outcome in test-index order (RE / TLE / WA). When run is nil the sandbox is unavailable, so the grade abstains with OfficialHarnessAvailable=false and Pass=false — never a fabricated pass. A problem with no test cases likewise abstains: there is nothing to witness.

type NormalizeOptions

type NormalizeOptions struct {
	Release   string
	Scenario  Scenario
	DatasetID string
	Revision  string
	Split     string
	FetchedAt string
	Model     string
}

NormalizeOptions pins the identity a normalized Suite records. Release and the provenance dataset id are required; Scenario defaults to codegeneration (the scenario code_generation_lite carries). Revision defaults to the resolved release when empty. FetchedAt is injected, never read from the wall clock here, so a normalized suite is byte-for-byte reproducible in a test or re-run.

type OfficialHarness

type OfficialHarness struct {
	Required  bool   `json:"required"`
	Available bool   `json:"available"`
	Reason    string `json:"reason"`
}

OfficialHarness mirrors terminalbench's official-harness gate (#2113): Required is always true for LiveCodeBench — only the official lcb_runner checker grading the exact saved generations is result-bearing — and Available reports whether that grading actually backs this report. A local-only run carries Available=false, and Validate refuses result_claim_allowed=true unless Required and Available are both true, so forging the evidence class alone can never flip the flag.

type OfficialRunContract

type OfficialRunContract struct {
	Schema              string                   `json:"schema"`
	GeneratedAt         string                   `json:"generated_at,omitempty"`
	Issue               string                   `json:"issue,omitempty"`
	Benchmark           string                   `json:"benchmark"`
	Status              string                   `json:"status"`
	EvidenceClass       string                   `json:"evidence_class"`
	ClaimBoundary       string                   `json:"claim_boundary"`
	Constants           ContractConstants        `json:"constants"`
	ProblemSelection    ContractProblemSelection `json:"problem_selection"`
	Arms                []ContractArm            `json:"arms"`
	Grading             ContractGrading          `json:"grading"`
	UpstreamRefs        []ContractUpstreamRef    `json:"upstream_refs"`
	Gates               []ContractGate           `json:"gates"`
	CompareMetrics      []string                 `json:"compare_metrics"`
	RequiredBeforeClaim []string                 `json:"required_before_claim"`
	ResultClaimAllowed  bool                     `json:"result_claim_allowed"`

	// PureKernel reports whether the recorded engine is fak's own in-kernel
	// decode, i.e. no external serving engine sits in the path.
	PureKernel bool `json:"pure_kernel"`
	// PureKernelResultStatus is `pending GPU run` for a pure-kernel arm and
	// empty otherwise. It is the machine-readable form of the honest fence: a
	// pure-kernel LCB codegen pass rate exists only once a real GPU run
	// through the in-kernel path is graded by the official evaluator (#2107).
	PureKernelResultStatus string `json:"pure_kernel_result_status,omitempty"`
}

OfficialRunContract is the result-claim-gated run contract. It mirrors the terminalbench official-run contract shape but is LiveCodeBench-native: two arms plus a single official grading authority.

func BuildOfficialRunContract

func BuildOfficialRunContract(in OfficialRunContractInput) OfficialRunContract

BuildOfficialRunContract builds the result-claim-gated contract. It is pure: it performs no run, touches no network, and always sets ResultClaimAllowed=false — only the official evaluator can ever back a score.

type OfficialRunContractInput

type OfficialRunContractInput struct {
	GeneratedAt     string
	Issue           string
	Suite           *Suite
	SuitePath       string
	ReleaseSelector string
	Scenario        Scenario
	StartDate       string
	EndDate         string
	Model           string
	Engine          string
	ServingBackend  string
	Gateway         string
	RunDir          string
}

OfficialRunContractInput carries the campaign coordinates the contract pins. Suite is optional: when supplied, its question_ids for the chosen scenario become the candidate problem selection so raw and fak arms provably score the same problems. No host/channel/token/URL belongs in here — the gateway is a local base URL only; secrets stay in gitignored config (#3059).

type Preflight

type Preflight struct {
	Schema             string          `json:"schema"`
	GeneratedAt        string          `json:"generated_at,omitempty"`
	Issue              string          `json:"issue,omitempty"`
	Status             string          `json:"status"`
	Gates              []PreflightGate `json:"gates"`
	BlockingReasons    []string        `json:"blocking_reasons"`
	NextAction         string          `json:"next_action"`
	ResultClaimAllowed bool            `json:"result_claim_allowed"`
	ClaimBoundary      string          `json:"claim_boundary"`
}

Preflight is the result-claim-gated host-readiness artifact.

func BuildPreflight

func BuildPreflight(in PreflightInput) Preflight

BuildPreflight classifies a probe into the gated preflight artifact. It never allows a result claim: a preflight measures host readiness, not the benchmark.

type PreflightGate

type PreflightGate struct {
	Name   string `json:"name"`
	OK     bool   `json:"ok"`
	Detail string `json:"detail,omitempty"`
}

PreflightGate is one host capability the run needs.

type PreflightInput

type PreflightInput struct {
	GeneratedAt string
	Issue       string
	Probe       PreflightProbe
}

PreflightInput carries the probe plus the campaign context the artifact records.

type PreflightProbe

type PreflightProbe struct {
	UvPresent     bool
	UvVersion     string
	PythonPresent bool
	// PythonVersion must report a 3.11.x string to satisfy the gate; the
	// official LiveCodeBench runner pins Python 3.11.
	PythonVersion string

	DatasetChecked   bool
	DatasetReachable bool
	DatasetURL       string

	GatewayChecked   bool
	GatewayReachable bool
	GatewayURL       string

	SandboxAvailable bool
	SandboxDetail    string
}

PreflightProbe is the observed host state. The caller (cmd/livecodebench) probes the live host; the classifier below is pure so it is fully unit-testable without uv, Python, a network, or a sandbox.

type Problem

type Problem struct {
	QuestionID   string     `json:"question_id"`
	Scenario     Scenario   `json:"scenario"`
	Platform     string     `json:"platform,omitempty"`
	Difficulty   string     `json:"difficulty,omitempty"`
	ContestDate  string     `json:"contest_date,omitempty"`
	Prompt       string     `json:"prompt"`
	StarterCode  string     `json:"starter_code,omitempty"`
	PublicTests  []TestCase `json:"public_test_cases,omitempty"`
	PrivateTests []TestCase `json:"private_test_cases,omitempty"`
}

Problem carries the LCB-native fields for one benchmark question.

type ProblemVerdict

type ProblemVerdict struct {
	QuestionID string   `json:"question_id"`
	Scenario   Scenario `json:"scenario,omitempty"`
	Arm        string   `json:"arm,omitempty"`
	Verdict    string   `json:"verdict"`
	EvidenceID string   `json:"evidence_id"`
}

ProblemVerdict is one per-problem row of a report: it links a problem's question_id to its verdict and the evidence id that backs that verdict, so a report reader can trace every row to gradeable evidence. Arm is empty when a row is not attributed to a specific raw|fak arm (e.g. a local ungraded scaffold before either arm has run).

func ProblemRowsFromSuite

func ProblemRowsFromSuite(s Suite) []ProblemVerdict

ProblemRowsFromSuite projects each suite problem into an ungraded per-problem verdict row, preserving suite order. No grading has happened, so every row's verdict is VerdictUngraded and its evidence id names the local, ungraded evidence class for that question. A graded run replaces these rows with real pass|fail verdicts and official-grading evidence ids.

type Provenance

type Provenance struct {
	DatasetID       string `json:"dataset_id"`                  // HF dataset, e.g. "livecodebench/code_generation_lite"
	Revision        string `json:"revision"`                    // dataset config/revision the rows were read at (e.g. "release_v2")
	Split           string `json:"split,omitempty"`             // dataset split, e.g. "test"
	FetchedAt       string `json:"fetched_at,omitempty"`        // RFC3339; injected by the caller, never read from the wall clock in pure code
	ProblemCount    int    `json:"problem_count"`               // number of problems the suite carries; must equal len(Problems)
	ContestDateFrom string `json:"contest_date_from,omitempty"` // YYYY-MM-DD, oldest contest_date in the suite
	ContestDateTo   string `json:"contest_date_to,omitempty"`   // YYYY-MM-DD, newest contest_date in the suite
}

Provenance pins where a normalized Suite came from, so a reader can trace a result back to an exact dataset version and problem window instead of an implicit "the benchmark". A normalized suite MUST carry it (see Suite.Validate) — an unsourced suite cannot back an honest result, and its ProblemCount must equal the number of problems it ships so the header cannot drift from the body.

type RawArmConfig

type RawArmConfig struct {
	Model       string  // model id recorded in the report and sent on each request
	Endpoint    string  // gateway base URL (…/v1) the completions are POSTed to
	N           int     // samples per problem (mirrors lcb_runner -n)
	Temperature float64 // sampling temperature recorded and sent
	Seed        int64   // sampling seed sent when nonzero and recorded; 0 = provider default
	Concurrency int     // max in-flight requests (mirrors closed-API --multiprocess)
	MaxRetries  int     // per-sample retry budget (#2106); a failed sample is retried and counted, never silently dropped
}

RawArmConfig pins the raw arm's run identity and its fan-out width. The sampling fields mirror SamplingConfig (#2106) — build it via SamplingConfig.ArmConfig so both arms share one sampling identity.

type RawArmProblem

type RawArmProblem struct {
	QuestionID   string   `json:"question_id"`
	PromptSHA256 string   `json:"prompt_sha256,omitempty"`
	Completions  []string `json:"completions"`
}

RawArmProblem holds the n completions collected for one problem, in sample order. PromptSHA256 is the hash of the exact rendered prompt this arm sent, so a cross-arm comparison can assert SamePromptHash from the artifacts alone (#2105).

type RawArmReport

type RawArmReport struct {
	Arm         string          `json:"arm"` // always "raw"
	Model       string          `json:"model"`
	Endpoint    string          `json:"endpoint"`
	N           int             `json:"n"`
	Temperature float64         `json:"temperature"`
	Seed        int64           `json:"seed,omitempty"` // 0 = provider default, omitted
	Concurrency int             `json:"concurrency"`
	MaxRetries  int             `json:"max_retries"`       // per-sample retry budget the run honored (#2106)
	Release     string          `json:"release,omitempty"` // dataset release the suite pinned (stamped by RunRawArmCached)
	Problems    []RawArmProblem `json:"problems"`
	Usage       RawArmUsage     `json:"usage"`
}

RawArmReport is the machine-readable result of the raw arm: the run identity (model / endpoint / n / temperature) plus per-problem completions and the folded provider-cache-aware token usage.

func RunRawArm

func RunRawArm(ctx context.Context, cfg RawArmConfig, problems []Problem, sample RawArmSampler) (RawArmReport, error)

RunRawArm fans the sampler out over every (problem, sample) pair with at most cfg.Concurrency requests in flight, then assembles a deterministic report. A failed sample is retried up to cfg.MaxRetries times, each retry counted in Usage.Retries (#2106); a sample that exhausts its budget aborts the run (the context is cancelled so in-flight siblings stop) and is returned as an error naming the problem and attempt count — a failure is never silently dropped. Completions are ordered by problem then sample index regardless of completion order, so the report is reproducible.

type RawArmSampler

type RawArmSampler func(ctx context.Context, p Problem, i int) (content string, u RawSampleUsage, err error)

RawArmSampler produces ONE completion for problem p at sample index i. The real implementation POSTs to the gateway's /v1/chat/completions and normalizes the provider-relayed usage into RawSampleUsage; tests inject a deterministic stub.

type RawArmUsage

type RawArmUsage struct {
	Samples            int `json:"samples"`
	PromptTokens       int `json:"prompt_tokens"`
	CompletionTokens   int `json:"completion_tokens"`
	CachedPromptTokens int `json:"cached_prompt_tokens"`
	Retries            int `json:"retries"` // failed sample attempts that were retried (#2106) — counted, never silently dropped
}

RawArmUsage is the run's token accounting folded across every sample. CachedPromptTokens is summed from the sampler-normalized RawSampleUsage.CachedPromptTokens so a provider-cache hit is counted the same way on OpenAI-compatible, DeepSeek, and Anthropic-shaped usage.

type RawCachedResult

type RawCachedResult struct {
	Report  RawArmReport
	Stats   CacheStats
	Resumed int // suite problems carried complete from the prior report
}

RawCachedResult is what a cached/resumable raw-arm run reports beyond the artifact itself. Stats counts ONLY genuine cache lookups (the honest denominator from cache.go); problems carried from a prior report via --continue-existing are counted in Resumed, never as cache hits.

func RunRawArmCached

func RunRawArmCached(ctx context.Context, cfg RawArmConfig, release string, problems []Problem, sample RawArmSampler, cache GenCache, prior *RawArmReport) (RawCachedResult, error)

RunRawArmCached runs the raw arm with #2108's reuse semantics layered around RunRawArm:

• prior (from --continue-existing) carries every problem the existing report already completed with exactly n completions; those problems are never regenerated (PendingWork guarantees no duplicates). The prior run's identity must match cfg — resuming under a different model, n, temperature, or seed is refused rather than silently mixed.
• cache (from --use-cache) is consulted once per still-pending problem; a hit with exactly n completions is reused, anything else is a miss and the problem is regenerated. Freshly generated completions are Put back so the NEXT run can reuse them — including runs interrupted before writing a report, since Put happens per problem, not per run.
• the merged report lists every suite problem in suite order. Its usage sums the prior report's usage with this run's generation usage; completions served from the cache add zero, because this run spent zero tokens on them.

With cache == nil and prior == nil it is exactly RunRawArm.

type RawSampleUsage

type RawSampleUsage struct {
	PromptTokens       int
	CompletionTokens   int
	CachedPromptTokens int
}

RawSampleUsage is the token accounting for ONE sample, already normalized by the sampler: CachedPromptTokens must be folded from the provider's own shape (the gateway sampler uses agent.Usage.CachedPromptTokens()) so a provider-cache hit counts the same on OpenAI-compatible, DeepSeek, and Anthropic-shaped usage.

type ReleaseHeader

type ReleaseHeader struct {
	Selection ReleaseSelection `json:"release"`
	Problems  int              `json:"problems"`
}

ReleaseHeader pins a resolved release alongside the number of problems scored, so a report reader knows exactly which dataset version(s) a result covers and over how many problems — never an implicit "latest" a reader cannot verify.

func PinRelease

func PinRelease(selector string, problems int) (ReleaseHeader, error)

PinRelease resolves the selector and stamps it against a problem count for a report header, so the resolved release and the count it was scored over are recorded together and cannot drift.

type ReleaseSelection

type ReleaseSelection struct {
	Selector string   `json:"selector"`
	Releases []string `json:"releases"`
	Resolved string   `json:"resolved"`
}

ReleaseSelection is the resolved outcome of a --release-version selector: the selector echoed back, the concrete releases it expands to (a single release or an inclusive range), and the newest release the selection covers. Releases is always non-empty and ordered oldest-to-newest.

func ResolveRelease

func ResolveRelease(selector string) (ReleaseSelection, error)

ResolveRelease parses an lcb_runner-style --release-version selector, mirroring upstream so results are comparable across dataset versions:

• "" or "release_latest": the newest known release, recorded explicitly.
• "release_vN" (v1..v6): that single release.
• "vA_vB" (A <= B): the inclusive range release_vA .. release_vB.

An unknown version, a malformed range, or a reversed range is a clear error rather than a silent default.

type RepairSource

type RepairSource struct {
	QuestionID    string `json:"question_id"`
	SourceSamples int    `json:"source_samples"` // n: source codegen generations for this problem
	SourceCorrect int    `json:"source_correct"` // c: source generations that passed grading
	Fixed         int    `json:"fixed"`          // of the (n-c) failing, how many the n=1 repair fixed
}

RepairSource is the graded codegen result self-repair builds on, for one problem: SourceSamples (= upstream --codegen_n) generations of which SourceCorrect passed grading, and Fixed of the (SourceSamples-SourceCorrect) failing generations that the n=1 repair pass turned into passes. Self-repair cannot run without this source — the repair prompt feeds a prior wrong generation and its failing-test feedback, so a codegen run must have produced and graded those generations first.

type Report

type Report struct {
	Schema         string `json:"schema"`
	GeneratedAt    string `json:"generated_at"`
	Benchmark      string `json:"benchmark"`
	Model          string `json:"model,omitempty"`
	ReleaseVersion string `json:"release_version"`
	StartDate      string `json:"start_date,omitempty"`
	EndDate        string `json:"end_date,omitempty"`
	EvidenceClass  string `json:"evidence_class,omitempty"`
	// WindowScore carries the contamination-free date-windowed rate alongside the
	// full-set rate when a run scopes results to a contest-date window (see
	// WindowedPassAtK); nil when the run was not date-windowed.
	WindowScore *WindowScores `json:"window_score,omitempty"`
	Arms        []ArmResult   `json:"arms,omitempty"`
	// Problems are the per-problem verdict rows: each links a question_id to its
	// verdict and the evidence id that backs it, so every row in a rendered
	// report traces to gradeable evidence. Empty for a summary-only report.
	Problems              []ProblemVerdict `json:"problems,omitempty"`
	Summary               Summary          `json:"summary"`
	OfficialHarness       OfficialHarness  `json:"official_harness"`
	PromotionRequirements []string         `json:"promotion_requirements,omitempty"`
	ResultClaimAllowed    bool             `json:"result_claim_allowed"`
	ClaimBoundary         string           `json:"claim_boundary,omitempty"`
}

Report is a run report over a Suite. Pass-rate fields cannot be carried without an evidence class, and ResultClaimAllowed cannot be true unless the evidence class is EvidenceOfficialLCBRunner — see Validate.

func NewReport

func NewReport(s Suite, generatedAt time.Time) Report

NewReport scaffolds an honest, unpromoted report over a suite: local evidence only, no result claim, and the promotion requirements spelled out.

func (*Report) MarkOfficiallyGraded

func (r *Report) MarkOfficiallyGraded() error

MarkOfficiallyGraded promotes a report after the official lcb_runner evaluator has graded the exact saved generations: it stamps the official evidence class and flips result_claim_allowed, then re-validates the whole report so a promotion can never produce an invalid artifact. It refuses a report with no graded arm results — with nothing graded there is nothing to promote, and the report stays at its local evidence class.

func (Report) Validate

func (r Report) Validate() error

type RunConfig

type RunConfig struct {
	Model          string   `json:"model"`
	Scenario       Scenario `json:"scenario"`
	Evaluate       bool     `json:"evaluate"`
	ReleaseVersion string   `json:"release_version"`
	N              int      `json:"n"`
	Temperature    float64  `json:"temperature"`
	UseCache       bool     `json:"use_cache"`
}

RunConfig captures the lcb_runner.runner.main-parity knobs a fak-native LiveCodeBench run is invoked with. Each field mirrors exactly one upstream flag so a fak run and a raw `python -m lcb_runner.runner.main` run are configured the same way:

Model       -> --model
Scenario    -> --scenario
Evaluate    -> --evaluate
Release     -> --release_version
N           -> -n
Temperature -> --temperature
UseCache    -> --use_cache

type RunReport

type RunReport struct {
	Schema             string           `json:"schema"`
	Config             RunConfig        `json:"config"`
	ReleaseVersion     string           `json:"release_version"`
	StartDate          string           `json:"start_date"`
	EndDate            string           `json:"end_date"`
	Questions          int              `json:"questions"`
	Scenarios          []ScenarioReport `json:"scenarios"`
	Evaluated          bool             `json:"evaluated"`
	ResultClaimAllowed bool             `json:"result_claim_allowed"`
	EvidenceClass      string           `json:"evidence_class"`
	PromotionRequired  []string         `json:"promotion_required"`
}

RunReport is the end-to-end artifact a fixture run writes. It records the resolved lcb_runner-parity config and the per-scenario question breakdown of the committed fixture. It is a fixture smoke, never a claimable score: ResultClaimAllowed stays false and promotion still requires the official lcb_runner grading (the same honesty fence SmokeReport carries).

func BuildRunReport

func BuildRunReport(f Fixture, cfg RunConfig) RunReport

BuildRunReport runs the committed fixture end-to-end under cfg: it scopes the fixture to the requested scenario (as lcb_runner runs one scenario per invocation) and emits a result-claim-gated report. The release defaults to the fixture's pinned release when cfg leaves it empty.

type SampleTally

type SampleTally struct {
	QuestionID string
	Samples    int // n: total generations for this problem
	Correct    int // c: generations that passed grading
}

SampleTally is one problem's per-problem sample count: Samples generations of which Correct passed the grader. QuestionID is carried only for error context.

type SamplingConfig

type SamplingConfig struct {
	N           int     // samples per problem (upstream -n)
	Temperature float64 // sampling temperature (upstream --temperature)
	Seed        int64   // sampling seed sent when nonzero; 0 = provider default
	Concurrency int     // max in-flight requests (rate limit toward the gateway)
	MaxRetries  int     // per-sample retry budget before the run aborts
}

SamplingConfig is the sampling identity shared by both arms. Both arms bind it into their run config through RawArmConfig so the shared knobs cannot drift between them.

func DefaultSampling

func DefaultSampling() SamplingConfig

DefaultSampling returns the upstream lcb_runner sampling defaults.

func (SamplingConfig) ArmConfig

func (s SamplingConfig) ArmConfig(model, endpoint string) RawArmConfig

ArmConfig binds this sampling identity to one arm's run identity (model + endpoint). RunFakArm takes the same RawArmConfig, so a config built here is the single source of truth for both arms of an A/B run.

func (*SamplingConfig) RegisterFlags

func (s *SamplingConfig) RegisterFlags(fs *flag.FlagSet)

RegisterFlags binds the shared sampling flag surface onto fs, defaulting to the receiver's current values. Both arm CLIs register their sampling flags through this one method, so the surface cannot drift between arms.

func (SamplingConfig) Validate

func (s SamplingConfig) Validate() error

Validate refuses a sampling config no arm should run with, so a bad flag value is caught before any tokens are spent.

type Scenario

type Scenario string

Scenario is one of the four upstream LiveCodeBench scenarios.

const (
	ScenarioCodeGeneration       Scenario = "codegeneration"
	ScenarioSelfRepair           Scenario = "selfrepair"
	ScenarioTestOutputPrediction Scenario = "testoutputprediction"
	ScenarioCodeExecution        Scenario = "codeexecution"
)

type ScenarioReport

type ScenarioReport struct {
	Scenario  string `json:"scenario"`
	Questions int    `json:"questions"`
}

type SelfRepairDelta

type SelfRepairDelta struct {
	Schema             string   `json:"schema"`
	Scenario           Scenario `json:"scenario"`
	Model              string   `json:"model,omitempty"`
	CodegenN           int      `json:"codegen_n"`
	RepairN            int      `json:"repair_n"`
	Problems           int      `json:"problems"`
	SourcePassAt1      float64  `json:"source_pass_at_1"`
	RepairedPassAt1    float64  `json:"repaired_pass_at_1"`
	Delta              float64  `json:"delta"`
	EvidenceClass      string   `json:"evidence_class"`
	ResultClaimAllowed bool     `json:"result_claim_allowed"`
}

SelfRepairDelta is the machine-readable result of a self-repair run: the source codegen pass@1, the post-repair pass@1, and their difference, plus the run identity the scenario demands (the source codegen_n and the fixed repair_n=1).

func BuildSelfRepairDelta

func BuildSelfRepairDelta(model string, codegenN int, sources []RepairSource) (SelfRepairDelta, error)

BuildSelfRepairDelta scores a self-repair run against its source codegen run. It REFUSES (clear error) without a codegen source — an empty source set, or a problem with no source generations, means there is nothing to repair. The repair pass is defined at n=1 (SelfRepairRepairN) and recorded as such. The source pass@1 is the mean pass@1 over the source (SourceSamples, SourceCorrect) tallies; the repaired pass@1 is the mean pass@1 over the SAME sample counts with correct = SourceCorrect + Fixed; the delta is repaired minus source.

type Suite

type Suite struct {
	Schema         string     `json:"schema"`
	Benchmark      string     `json:"benchmark"`
	Model          string     `json:"model,omitempty"`
	ReleaseVersion string     `json:"release_version"`
	Provenance     Provenance `json:"provenance"`
	Problems       []Problem  `json:"problems"`
}

Suite is a normalized, release-pinned LiveCodeBench problem set.

func LoadSuite

func LoadSuite(r io.Reader) (Suite, error)

func LoadSuiteFile

func LoadSuiteFile(path string) (Suite, error)

func Normalize

func Normalize(ups []UpstreamProblem, opts NormalizeOptions) (Suite, error)

Normalize maps upstream LiveCodeBench rows into a release-pinned, sourced Suite. It normalizes contest_date to YYYY-MM-DD, lowercases platform and difficulty to the Suite's enums, parses the upstream public_test_cases JSON string into TestCases, and stamps a provenance header (dataset, revision, problem count, contest-date range). The returned Suite is Validated, so a normalized suite is always loadable and always carries its provenance.

func (Suite) Validate

func (s Suite) Validate() error

type Summary

type Summary struct {
	Problems  int              `json:"problems"`
	Graded    int              `json:"graded"`
	Scenarios []ScenarioReport `json:"scenarios,omitempty"`
}

Summary folds the suite shape a report ran over.

type TestCase

type TestCase struct {
	Input    string `json:"input"`
	Output   string `json:"output"`
	TestType string `json:"testtype,omitempty"`
}

TestCase mirrors the upstream test-case shape (testtype: stdin | functional).

type TestOutcome

type TestOutcome struct {
	Index   int    `json:"index"`
	Verdict string `json:"verdict"`
	Detail  string `json:"detail,omitempty"`
}

TestOutcome is one test case's execution result.

type TestOutputCase

type TestOutputCase struct {
	Input     string `json:"input"`
	Expected  string `json:"expected"`
	Predicted string `json:"predicted"`
	Correct   bool   `json:"correct"`
}

TestOutputCase is the graded result for one problem input: the input, its expected output, the model's predicted output, and whether they matched under GradeTestOutput.

type TestOutputProblem

type TestOutputProblem struct {
	QuestionID string           `json:"question_id"`
	Cases      []TestOutputCase `json:"cases"`
	Correct    bool             `json:"correct"`
}

TestOutputProblem is the per-problem prediction result: every input's predicted vs expected output and whether the model got ALL of them right. A problem is Correct only when every one of its inputs is predicted correctly — a partially-right multi-input problem is a miss, matching upstream all-or-nothing scoring.

func GradeTestOutputProblem

func GradeTestOutputProblem(p Problem, predicted []string) (TestOutputProblem, error)

GradeTestOutputProblem grades one problem's predictions: predicted[i] is the model's output for the problem's i-th public test-case input, graded against that case's expected Output. It refuses when the problem carries no test cases (nothing to predict) or when the prediction count does not match the input count (a prediction is required for every input — a short slice would silently score missing inputs as passes). The returned problem is Correct only when every input matched.

type TestOutputSummary

type TestOutputSummary struct {
	Schema             string              `json:"schema"`
	Scenario           Scenario            `json:"scenario"`
	Model              string              `json:"model,omitempty"`
	Problems           []TestOutputProblem `json:"problems"`
	Correct            int                 `json:"correct"`
	Accuracy           float64             `json:"accuracy"`
	EvidenceClass      string              `json:"evidence_class"`
	ResultClaimAllowed bool                `json:"result_claim_allowed"`
}

TestOutputSummary is the machine-readable result of a test-output-prediction run: the per-problem correctness rows and the summary accuracy (fraction of problems all of whose inputs were predicted correctly), plus the evidence gate.

func BuildTestOutputSummary

func BuildTestOutputSummary(model string, problems []TestOutputProblem) (TestOutputSummary, error)

BuildTestOutputSummary aggregates graded per-problem predictions into a run summary: the accuracy is the fraction of problems all of whose inputs the model predicted correctly. It refuses an empty set (no problems graded) or a problem with a blank question_id. Evidence is gated — ResultClaimAllowed stays false until the official lcb_runner grades the same predictions.

type UpstreamProblem

type UpstreamProblem struct {
	QuestionID      string `json:"question_id"`
	QuestionTitle   string `json:"question_title"`
	QuestionContent string `json:"question_content"`
	Platform        string `json:"platform"`
	Difficulty      string `json:"difficulty"`
	ContestDate     string `json:"contest_date"`
	StarterCode     string `json:"starter_code"`
	PublicTestCases string `json:"public_test_cases"`
}

UpstreamProblem is the subset of a LiveCodeBench code_generation dataset row (the HuggingFace `livecodebench/code_generation_lite` shape) that a fak Suite needs. Upstream stores public_test_cases as a JSON-encoded STRING and contest_date as an RFC3339-ish timestamp; Normalize turns both into the Suite's native shapes. Fields we do not consume (private_test_cases, which upstream ships zlib+base64-encoded, and metadata) are intentionally omitted: the fak arm generates from the prompt, and grading runs through lcb_runner.

func ParseUpstreamRows

func ParseUpstreamRows(data []byte) ([]UpstreamProblem, error)

ParseUpstreamRows decodes upstream problems from either a bare JSON array of UpstreamProblem or a HuggingFace datasets-server {"rows":[{"row":...}]} envelope, dispatching on the first non-space byte.

type WindowScores

type WindowScores struct {
	StartDate        string  `json:"start_date,omitempty"`
	EndDate          string  `json:"end_date"`
	K                int     `json:"k"`
	TotalProblems    int     `json:"total_problems"`
	WindowedProblems int     `json:"windowed_problems"`
	FullPassRate     float64 `json:"full_pass_rate"`
	WindowedPassRate float64 `json:"windowed_pass_rate"`
}

WindowScores is the reportable contamination-window result: pass@k over the full problem set AND over the windowed subset, the window bounds, and the two problem counts. WindowedPassRate is scored over strictly the in-window problems, so a problem outside the window never moves it.

func WindowedPassAtK

func WindowedPassAtK(problems []Problem, tallies []SampleTally, k int, w DateWindow) (WindowScores, error)

WindowedPassAtK joins each per-problem tally to its problem by question_id, then scores pass@k twice: over all problems (the full-set rate) and over only those in the window (the contamination-free rate). It errors when a tally names a question_id absent from problems (the two inputs must describe the same run), when problems carries a duplicate question_id (an ambiguous join), or when pass@k is invalid. When no problem falls in the window the windowed rate is 0 over 0 problems — an honest empty window, not an error.