bench

Documentation

Overview

Package bench loads benchmark scenarios from evals/ and materializes their fixtures into sandbox git repositories.

Index

• Constants
• func Materialize(s *Scenario, dir string) error
• func ScoreCell(tier string, r RunReport, expected int) string
• type CheckResult
  • func Check(s *Scenario, res Result) (CheckResult, error)
  • func (c CheckResult) HitCount() int
• type Config
  • func LoadConfig(path string) (Config, error)
  • func (c Config) Model(id harness.ID) (string, error)
• type Expectation
• type HarnessReport
• type Report
  • func (r Report) Markdown() string
• type Result
• type RunReport
• type Runner
  • func (r Runner) Run(ctx context.Context, s *Scenario, skill SkillVersion) (Result, error)
• type Scenario
  • func LoadScenario(dir string) (*Scenario, error)
  • func LoadScenarios(evalsDir, skill string) ([]*Scenario, error)
  • func (s *Scenario) ExpectedHits() int
• type ScenarioReport
• type SkillVersion
  • func LoadVersions(root, skill string) ([]SkillVersion, error)

Constants

const (
	DefaultBranch = "main"
	WorkBranch    = "change"
)

Branch names of the materialized fixture repo.

const (
	TierPlantedDefect = "planted-defect"
	TierStructural    = "structural"
	TierSmoke         = "smoke"
)

Check tiers (FR-10).

const (
	StyleReport = "report"
	StyleApply  = "apply"
)

Skill output styles for planted-defect checking.

const (
	LabelOld = "old"
	LabelNew = "new"
)

Version labels used in run headers and reports.

const DefaultTimeout = 5 * time.Minute

DefaultTimeout bounds one harness invocation.

Functions

func Materialize

func Materialize(s *Scenario, dir string) error

Materialize turns a scenario into a git repository at dir (an existing empty directory): the base/ tree committed on DefaultBranch, then the change/ tree overlaid and committed on WorkBranch, which is left checked out. Host and system git config are ignored so runs are reproducible across machines.

func ScoreCell

func ScoreCell(tier string, r RunReport, expected int) string

ScoreCell renders one run's score for a tier — the single wording shared by the run stream and the report table, so the two can't drift.

Types

type CheckResult

type CheckResult struct {
	// Hits is parallel to Scenario.Expectations (planted-defect) or
	// Scenario.Elements (structural); smoke has a single pass/fail entry.
	Hits []bool
	// Extras are findings matching no expectation — counted and listed but
	// never scored (FR-11). Report style: output lines mentioning a fixture
	// file that no expectation accounts for. Apply style: changed files no
	// expectation names.
	Extras []string
}

CheckResult scores one run against a scenario's expectations.

func Check

func Check(s *Scenario, res Result) (CheckResult, error)

Check scores a run deterministically per the scenario's tier (FR-10).

func (CheckResult) HitCount

func (c CheckResult) HitCount() int

HitCount is the number of planted defects the run found.

type Config

type Config struct {
	Models map[string]string `yaml:"models"`
}

Config is the checked-in bench configuration (evals/bench.yaml): the model each harness is pinned to, so runs are reproducible without flags.

func LoadConfig

func LoadConfig(path string) (Config, error)

LoadConfig reads and validates the bench config at path.

func (Config) Model

func (c Config) Model(id harness.ID) (string, error)

Model returns the pinned model for a harness.

type Expectation

type Expectation struct {
	File     string   `yaml:"file"`
	Keywords []string `yaml:"keywords"`
	Anchors  []string `yaml:"anchors"`
}

Expectation is one planted defect. Report-style skills hit by mentioning File plus any keyword; apply-style skills hit by changing the lines the anchors identify in the fixture.

type HarnessReport

type HarnessReport struct {
	Harness   string // display name
	Model     string // pinned model ID actually used
	Scenarios []ScenarioReport
}

HarnessReport is one harness's runs. Reports never compare across harnesses; each group stands alone.

type Report

type Report struct {
	Skill    string
	Command  string // exact reproduction command
	Baseline bool
	OldSHA   string // empty in baseline mode
	NewSHA   string
	Groups   []HarnessReport
}

Report is a completed bench invocation, renderable as PR-ready markdown (FR-12). It carries everything needed to re-run identically (NFR-3) and never a verdict on old vs new (FR-13) — interpretation belongs to the author and reviewer.

func (Report) Markdown

func (r Report) Markdown() string

Markdown renders the report in the pr-md format: per-harness per-run hit tables, aggregates, model IDs, the reproduction command, and raw transcripts collapsed in <details>.

type Result

type Result struct {
	Stdout string
	Stderr string
	Diff   string // post-run git diff of the sandbox, harness dirs excluded
	Err    error
}

Result captures one harness invocation. Err records a failed or timed-out invocation (FR-9: loud, never skipped); Stdout/Stderr are kept even then.

type RunReport

type RunReport struct {
	Failed  bool
	FailMsg string
	Checked bool // tier scoring ran
	Hits    int
	Extras  int
	Stdout  string
	Stderr  string
	Diff    string
}

RunReport is one harness invocation's outcome.

type Runner

type Runner struct {
	Harness harness.ID
	Model   string
	Timeout time.Duration // 0 means DefaultTimeout
}

Runner invokes one harness with a pinned model.

func (Runner) Run

func (r Runner) Run(ctx context.Context, s *Scenario, skill SkillVersion) (Result, error)

Run benches one skill version against one scenario: materialize the fixture into a fresh sandbox, install the version project-locally, invoke the harness headlessly in the sandbox, capture output and the post-run diff. The returned error is infrastructural (sandbox, git); harness failures land in Result.Err.

type Scenario

type Scenario struct {
	Name string // scenario directory name
	Dir  string // path to the scenario directory

	Task  string `yaml:"task"`
	Tier  string `yaml:"tier"`
	Style string `yaml:"style"`
	// Expectations drive planted-defect checking; Elements drive structural.
	Expectations []Expectation `yaml:"expectations"`
	Elements     []string      `yaml:"elements"`
}

Scenario is one benchmark case: a fixture repo (base/ + change/) plus the task prompt and expectations from expectations.yaml.

func LoadScenario

func LoadScenario(dir string) (*Scenario, error)

LoadScenario loads and validates a single scenario directory.

func LoadScenarios

func LoadScenarios(evalsDir, skill string) ([]*Scenario, error)

LoadScenarios loads every scenario for a skill under evalsDir, sorted by name. It fails when the skill has no scenario directory.

func (*Scenario) ExpectedHits

func (s *Scenario) ExpectedHits() int

ExpectedHits is the per-run hit denominator: planted defects or required elements; smoke has none.

type ScenarioReport

type ScenarioReport struct {
	Name         string
	Tier         string
	Expectations int         // per-run hit denominator: planted defects or required elements; 0 for smoke
	Old          []RunReport // empty in baseline mode
	New          []RunReport
}

ScenarioReport is one scenario's old and new run sets on one harness.

type SkillVersion

type SkillVersion struct {
	Name  string
	Label string // LabelOld or LabelNew
	SHA   string // git blob SHA of the version's SKILL.md, for reproducible reports (NFR-3)
	// Files is the skill's whole directory — SKILL.md plus any companions —
	// keyed by slash-separated path relative to skills/<Name>/.
	Files map[string][]byte
}

SkillVersion is one version of a skill under bench: the working tree ("new") or the main-branch content ("old").

func LoadVersions

func LoadVersions(root, skill string) ([]SkillVersion, error)

LoadVersions resolves the skill versions to bench in the repo at root: "old" from the main branch via git (never the embedded catalog, FR-3.a) and "new" from the working tree. Each version carries the skill's whole directory — SKILL.md plus any companions — so the sandbox install matches a real one. When the skill is absent on the main branch the slice holds only "new" — baseline mode (FR-3.c).