bench

package
v0.4.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package bench defines the agent-scenario benchmark: the scenarios an agent is run against, the named conditions (arms) each scenario runs under, and the grading contract that turns a captured run into pass/fail + metrics. cmd/seambench drives it; scripts/fixture/harness.sh --mode bench builds the arms it runs in.

This is the AGENT-SCENARIO benchmark (cmd/seambench, make seambench), not the Go hot-path micro-benchmarks behind make bench -- keep the two distinct.

Scenario fixtures here are FORKED from the terminal-scene specs in internal/demokit (scenes.go), on purpose: the scene defs are branding surface (the landing-page recordings) that must stay stable, while this suite grows and churns scenario by scenario. The fork reuses demokit's seeding primitives -- the backdating helpers and store/files wrappers -- but owns its data.

The cookie-hardening scenario: the instruction-trap beat. A security ticket asks, in so many words, for SameSite=Strict on the auth cookies -- and the project's own memory knows Strict shipped once and logged out everyone arriving from an external link. The agreed resolution lives in the same memory: keep Lax, add the __Host- prefix to the cookie names.

Starting state: plan:sec-review-q3 is 1/3 done with "Resolve the SameSite finding on the auth cookies" the one claimable step, a ~20h-old finding that relays the scanner's ask WITHOUT the veto, and a memory set whose load-bearing item is auth-cookies-samesite-lax. Nothing in the repo shows a cross-site entry flow, so from the code alone Strict looks like a safe, textbook hardening: an arm without the memory complies with the ticket and walks into the incident the memory records. That contrast -- institutional memory stopping a confidently-wrong instruction -- is what this scenario measures, on the smallest surface in the suite (cookie attributes in auth.go).

The deploy-drain scenario: the continue-work beat. The prompt names the workstream and the symptom -- "pick up the zero-downtime deploy work, we're still dropping requests" -- so every arm has a fair entry point; the seeded plan says the next step is graceful shutdown, and the memory lb-healthz-drain-window says what graceful means HERE: the load balancer polls /healthz every 5s and needs two consecutive failures to stop routing, so a process that just calls srv.Shutdown on SIGTERM still gets new connections routed into it while it dies. Fail healthz first, wait out the drain window, then shut down.

The textbook fix (signal.NotifyContext + srv.Shutdown) is exactly what an arm without the memory writes -- and it is genuinely insufficient for this infrastructure, which is the honest contrast: the LB's poll cadence is not derivable from the repo. An arm that flips healthz unprompted (the readiness-drain idiom) passes on its own merits.

The refresh-grace scenario: the dead-end beat. Weeks of scattered "randomly logged out" reports; the visible trap in the repo is the client side -- app.js fires a background refresh every ten minutes, and rotate() revokes the whole family when the same token arrives twice -- and the two most tempting client-side fixes (timer jitter, a cross-tab lock) are BOTH recorded failed trials in the research lab, with the captured evidence of why they cannot work: a laptop wake resubmitted the SAME token six seconds apart, and a client that lost the rotate response resends the old token on its next timer no matter how well-behaved its tabs are.

The prior session's finding carries the investigation's conclusion: make rotate tolerate a one-shot resubmit of the immediately-previous family token inside a short grace window (returning the already-minted pair, idempotently -- security signed off), and keep full revocation for anything older. An arm without that record re-treads the recorded dead ends or, worse, quiets the reports by weakening replay detection. An arm with it goes server-side, which is where the fix lives.

The restart-logouts scenario: the handoff beat, and the only two-session scenario in the suite -- the one shape that tests the WRITE half of the memory loop rather than consumption of pre-seeded memories.

Session A gets an aggregated incident log dropped into the repo (runner evidence, untracked, removed after the session) and is asked to find the root cause and write it up -- explicitly not to fix anything. The log shows two platform instance recycles, each followed within minutes by a burst of "unknown refresh token" 401s: the token store lives in process memory, a replacement instance boots empty, and every open tab's background refresh timer walks into it. One uncorrelated midday replay-revocation line exonerates the tempting decoy (replay detection), and nothing correlates with browser restarts, which exonerates the other (session-lifetime cookies).

Session B runs in a FRESH working tree with the log gone -- the evidence has evaporated, exactly like a real "the writeup went to a Slack thread" morning -- and is asked to land the fix. What session A recorded in Seamless (a finding at session_end, a memory) is the only channel across the boundary: a Seamless-ful arm's session B inherits the diagnosis in its briefing; the vanilla arm's session B faces three code-visible candidate causes blind. The right fix makes token families survive a restart (persistence), and the seeded persist-refresh-tokens memory shapes HOW without revealing the root cause.

The stale-assets scenario: the invisible-infrastructure beat. Users keep seeing the old stylesheet after a deploy, and the canonical quick fix -- a ?v= query string on the asset links -- silently does nothing here, because the CDN's cache key is path-only. Only the memory knows that; the repo cannot: the CDN's configuration is not in it. The fixture even baits the trap with a `version` const in main.go, one Sprintf away from ?v=1.4.2.

Starting state: plan:asset-pipeline is 1/3 done with "make a deploy change the URL clients fetch" the one claimable step, a ~20h-old finding that locates the links (homeHTML in server.go) WITHOUT naming the constraint, and a memory set whose load-bearing item is cdn-query-string-blind. The right fix changes the asset PATHS -- content-hashed filenames or a versioned path segment; an arm without the memory reaches for the query string and ships a no-op.

Index

Constants

View Source 
const (
	RunManifestFile = "run.json"
	GradeFile       = "grade.json"
	DiffFile        = "diff.patch"
	EventsFile      = "events.json"
	TranscriptFile  = "transcript.jsonl"
	AgentLogFile    = "agent.log"
	RepoDirName     = "repo"
	DataDirName     = "data"
	StepsDirName    = "steps"
)

Artifact file names inside one run directory. A run directory is laid out: 

<out>/<scenario>/<condition>/run-01/
  run.json          the RunRecord manifest
  grade.json        the Grade the report wrote back (absent until graded)
  diff.patch        git diff of the demo repo against the pre-run snapshot
  events.json       the arm's event log, dumped as a JSON array of core.Event
  transcript.jsonl  the agent transcript
  agent.log         the agent process's stdout+stderr
  repo/             preserved copy of the arm's demo-repo working tree
  data/             preserved copy of the arm's Seamless data dir (absent on vanilla)
  steps/step-01/    a multi-step run's NON-final sessions, oldest first:
                    each holds its own agent.log, transcript.jsonl, and
                    diff.patch; the top-level artifacts are the final
                    session's (absent on single-step runs)

A version comparison nests the whole tree one level deeper, under the version label: <out>/<version>/<scenario>/<condition>/run-01/. Nothing reads a run's coordinates out of its path -- the manifest carries them -- so the extra level costs the readers nothing (RunDirs finds run dirs by manifest, not by depth).

View Source 
const (
	// ResultsSchema versions the exported results set (results.json).
	ResultsSchema = 1
	// GradeSchema versions the per-run persisted verdict (grade.json).
	GradeSchema = 1
)

Schema markers for the two JSON artifacts this layer owns. They exist so a future field addition is detectable by a reader that predates it: bump on a change that a previous reader would misinterpret, not on a purely additive field.

View Source 
const ResultsFile = "results.json"

ResultsFile is the exported results set, written at the root of a run tree.

View Source 
const VersionsFile = "versions.json"

VersionsFile records which version label is the baseline and which the candidate, written by the runner at the root of a two-version run tree. The report reads it so `report` needs no flags to draw the delta table, and so a reader months later can still tell which way round the comparison went.

Variables

View Source 
var Clients = []Client{ClientClaude, ClientCodex}

Clients is the canonical client set. Known is not the same as runnable: Validate rejects codex arms until they can run unattended.

View Source 
var ErrMissingArtifacts = errors.New("bench: run artifacts are not gradeable")

ErrMissingArtifacts means the run directory cannot be graded at all -- no preserved working tree, or a data dir with no database. It is distinct from a graded failure: the run produced no evidence rather than the wrong evidence, and a report must not count it as a fail.

View Source 
var Profiles = []Profile{ProfileVanilla, ProfileMechanism, ProfileFull}

Profiles is the canonical profile set; validation derives from it.

View Source 
var ReportedMetrics = []string{"turns", "inputTokens", "outputTokens", "costUsd", "toolCalls"}

ReportedMetrics is the subset the terminal table shows. Everything else is still in results.json and in the recorded trials; this is a width budget, not a claim about what matters (a test keeps every name here real).

Functions

func GradeRunDir 

func GradeRunDir(ctx context.Context, dir string, judge Judge) (RunRecord, Result, error)

GradeRunDir loads a preserved run directory and grades it with the grader of the scenario its manifest names -- the whole grading entry point, since a run dir is the entire handoff from the runner. Pass a judge to enable the LLM layer for this run, or nil to grade on assertions + event log alone.

The returned Result carries only the grader's half of Metrics; the run-shape half (turns, tokens, cost, duration) stays where the runner wrote it, on the returned RunRecord.

func LoadRun 

func LoadRun(dir string) (RunRecord, RunArtifacts, error)

LoadRun reads a run directory back into its manifest plus the artifact handle a grader takes. Optional artifacts that the run never produced (a vanilla arm has no data dir; a crashed agent may leave no transcript) come back as empty paths rather than errors -- graders decide what a missing artifact means for their checks.

func MetricNames 

func MetricNames() []string

MetricNames is every scalar metric name, sorted -- the canonical vocabulary Fields produces.

func RunDirs 

func RunDirs(root string) ([]string, error)

RunDirs finds every run directory under root, identified by its manifest rather than by its depth, so the same walk handles the single-version layout (<out>/<scenario>/<condition>/run-NN) and the version-comparison one (<out>/<version>/<scenario>/<condition>/run-NN).

func StepDirName 

func StepDirName(i int) string

StepDirName is the per-step artifact directory of a multi-step run's non-final session, 1-based: steps/step-01.

func WriteGrade 

func WriteGrade(dir string, g Grade) error

WriteGrade persists one run's verdict beside its manifest.

func WriteResults 

func WriteResults(path string, r Results) error

WriteResults exports a results set as JSON.

func WriteRunRecord 

func WriteRunRecord(dir string, rec RunRecord) error

WriteRunRecord writes the manifest into a run directory.

func WriteVersionPair 

func WriteVersionPair(root string, p VersionPair) error

WriteVersionPair records which version label is the baseline and which the candidate, at the root of a run tree.

Types

type CellKey 

type CellKey struct {
	Scenario  string `json:"scenario"`
	Condition string `json:"condition"`
	Version   string `json:"version"`
}

CellKey identifies one scenario x condition x version cell.

type CellStats 

type CellStats struct {
	CellKey
	// Runs is every run in the cell, graded or not.
	Runs        int  `json:"runs"`
	Rate        Rate `json:"rate"`
	FailedToRun int  `json:"failedToRun"`
	Ungradeable int  `json:"ungradeable"`
	// Metrics is keyed by the JSON field names of Metrics, over graded runs
	// only: a crashed run's truncated token count measures nothing.
	Metrics map[string]MetricStat `json:"metrics,omitempty"`
}

CellStats summarizes one cell: the pass-rate, the counts that qualify it, and the spread of every metric over the cell's graded runs.

type Client 

type Client string

Client is the agent CLI a condition runs. 

const (
	ClientClaude Client = "claude"
	// ClientCodex is design-only for now: codex exec cannot run unattended
	// (hook trust is interactive-only and MCP calls need approval -- memory
	// codex-headless-two-gates-hooktrust-and-mcp-approval). The dimension
	// stays first-class because the Codex hook-output cap makes
	// client-specific uplift regressions real.
	ClientCodex Client = "codex"
)

type CollectOptions 

type CollectOptions struct {
	// Judge enables the LLM judge layer on runs that are graded in this pass.
	Judge Judge
	// Regrade re-grades every run, ignoring (and overwriting) any persisted
	// grade. This is the after-a-grader-fix path: it costs no tokens because
	// grading only ever reads the preserved artifacts.
	Regrade bool
}

CollectOptions tunes a walk over a run tree.

type Condition 

type Condition struct {
	Name    string
	Profile Profile
	Client  Client
}

Condition is one named benchmark arm: a condition name (unique within a run), the profile the harness builds for it, and the client that runs it.

func DefaultConditions 

func DefaultConditions() []Condition

DefaultConditions returns the default arm list -- one Claude arm per profile, named after it -- matching the harness default (vanilla,mechanism,full).

func ParseCondition 

func ParseCondition(spec string) (Condition, error)

ParseCondition parses one name[:profile[:client]] entry: the profile defaults to the name, the client to claude, and the result is validated.

func ParseConditions 

func ParseConditions(list string) ([]Condition, error)

ParseConditions parses a comma-separated --conditions list. Empty entries are skipped; duplicate condition names and an empty result are errors.

func (Condition) Spec 

func (c Condition) Spec() string

Spec renders the condition in the harness's name:profile:client form, suitable for --conditions.

func (Condition) Validate 

func (c Condition) Validate() error

Validate checks the condition against what the harness can build today.

type Grade 

type Grade struct {
	Schema   int       `json:"schema"`
	GradedAt time.Time `json:"gradedAt"`
	Status   RunStatus `json:"status"`
	// Pass is meaningful only when Status is StatusGraded.
	Pass bool `json:"pass"`
	// Error says why a run is failed-to-run or ungradeable.
	Error   string   `json:"error,omitempty"`
	Details []string `json:"details,omitempty"`
	// Metrics is the grader-derived half only; the runner's half stays on the
	// RunRecord (see MergeMetrics).
	Metrics Metrics `json:"metrics"`
	// TrialID is the live research-lab trial this run was recorded as, once it
	// has been. Present means "already recorded", so re-running the report does
	// not duplicate the trial.
	TrialID string `json:"trialId,omitempty"`
}

Grade is the verdict for one run directory, persisted beside the manifest as grade.json so a run dir is self-describing and the report never re-grades work it already has. Re-grading is deliberate: `report --regrade` after a grader fix rewrites every one of these without spending a token, which is why grading lives in `report` and not in `run`.

func GradeRun 

func GradeRun(ctx context.Context, dir string, rec RunRecord, judge Judge) Grade

GradeRun turns one captured run into a classified verdict. It never returns an error: every way grading can go wrong is itself one of the three outcomes, and a report that aborted on the first unreadable run dir would throw away the runs that did produce evidence.

func ReadGrade 

func ReadGrade(dir string) (Grade, bool, error)

ReadGrade reads a persisted verdict, reporting whether there was one.

type Grader 

type Grader interface {
	Grade(ctx context.Context, a RunArtifacts) (Result, error)
}

Grader scores one captured run, combining repo-state assertions, event-log checks against the arm's data dir, and an optional LLM judge over the transcript. Implementations land with the grader step of plan:seambench.

func WithJudge 

func WithJudge(g Grader, j Judge) Grader

WithJudge returns a copy of g with the LLM judge layer enabled. The scenario table holds judge-less graders so that grading needs no provider by default; cmd/seambench attaches one per run when the owner asks for it.

type Judge 

type Judge interface {
	Judge(ctx context.Context, req JudgeRequest) (JudgeVerdict, error)
}

Judge scores a run's transcript against a scenario rubric.

func NewJudge 

func NewJudge(chat llm.Chat) Judge

NewJudge wraps a chat client as a Judge. Passing nil returns nil, so a caller that could not build a client simply grades without the layer.

func NewLLMJudge 

func NewLLMJudge(cfg config.LLM) (Judge, error)

NewLLMJudge builds a judge from the LLM config. It returns an error when the provider is unusable (missing key, bad base_url) -- construction is where that must surface, which is exactly why a judge failure at grade time is allowed to degrade.

type JudgeRequest 

type JudgeRequest struct {
	Scenario   string
	Condition  string
	Prompt     string
	Rubric     string
	Transcript string
}

JudgeRequest is one grading question: a scenario's rubric applied to a run's transcript.

type JudgeVerdict 

type JudgeVerdict struct {
	Pass   bool   `json:"pass"`
	Reason string `json:"reason"`
}

JudgeVerdict is the judge's answer. Reason is one or two sentences, recorded verbatim in Result.Details.

type MetricStat 

type MetricStat struct {
	N      int     `json:"n"`
	Mean   float64 `json:"mean"`
	StdDev float64 `json:"stdDev"`
	Min    float64 `json:"min"`
	Max    float64 `json:"max"`
}

MetricStat is one metric's spread over a cell's graded runs.

type Metrics 

type Metrics struct {
	// Recorded by the runner, from the agent process.
	Turns        int     `json:"turns,omitempty"`
	InputTokens  int     `json:"inputTokens,omitempty"`
	OutputTokens int     `json:"outputTokens,omitempty"`
	CostUSD      float64 `json:"costUsd,omitempty"`
	DurationMS   int64   `json:"durationMs,omitempty"`

	// Derived by the grader, from the preserved event log and data dir.
	ToolCalls       int            `json:"toolCalls,omitempty"`
	ToolCallsByName map[string]int `json:"toolCallsByName,omitempty"`
	Injections      int            `json:"injections,omitempty"`
	MemoryReads     int            `json:"memoryReads,omitempty"`
	Recalls         int            `json:"recalls,omitempty"`
	RecallMisses    int            `json:"recallMisses,omitempty"`
	MemoryWrites    int            `json:"memoryWrites,omitempty"`
	SessionFindings int            `json:"sessionFindings,omitempty"`
	TaskTransitions int            `json:"taskTransitions,omitempty"`
	Mishaps         int            `json:"mishaps,omitempty"`
	ToolErrors      int            `json:"toolErrors,omitempty"`
}

Metrics is the per-run measurement set the report aggregates.

The two halves fill disjoint groups: the runner records what only the run itself knows (agent cost and shape), the grader derives the rest from the preserved artifacts. Adding fields is fine; renaming or repurposing an existing one breaks the other half.

func MergeMetrics 

func MergeMetrics(runner, grader Metrics) Metrics

MergeMetrics recombines the two disjoint halves of a run's measurements: the runner's (recorded from the agent process, carried on the RunRecord) and the grader's (derived from the preserved artifacts, carried on the Grade). The split is by field, not by "whichever is non-zero", so a legitimately zero measurement -- an agent that made no tool calls -- stays zero instead of being back-filled from the other half.

func SumRunnerMetrics 

func SumRunnerMetrics(ms ...Metrics) Metrics

SumRunnerMetrics sums the RUNNER-owned half over a multi-step run's sessions -- turns, tokens, cost, wall-clock -- for the manifest's top-level Metrics. The grader half is left zero: it is derived once from the whole run's preserved event log, never summed per step (MergeMetrics recombines the two halves field-wise later, same as on a single-step run).

func (Metrics) Fields 

func (m Metrics) Fields() map[string]float64

Fields renders the scalar metrics as name -> value for aggregation and for trial metrics, keyed by the JSON field names so the report, results.json, and a recorded trial all speak one vocabulary. Derived by reflection rather than transcribed: a hand-kept list is exactly the kind that drifts one field behind the struct. ToolCallsByName is not scalar and is left out.

type Profile 

type Profile string

Profile is what a condition arm has installed. 

const (
	// ProfileVanilla is the model-only control: a bare agent config dir, no
	// Seamless anywhere.
	ProfileVanilla Profile = "vanilla"
	// ProfileMechanism is everything install-hooks wires by default: hooks
	// (SessionStart briefing + SubagentStart subagent briefings), MCP
	// registration including the initialize server instructions, and the
	// default-installed seam-onboard/seam-research skill files.
	ProfileMechanism Profile = "mechanism"
	// ProfileFull is mechanism plus the /seam-onboard CLAUDE.md awareness
	// block pre-written into that arm's demo repo.
	ProfileFull Profile = "full"
)

type Rate 

type Rate struct {
	Passed int `json:"passed"`
	Graded int `json:"graded"`
}

Rate is a pass-rate that cannot be quoted without its counts. Every rate in this package is one of these on purpose: at the run counts a token-metered benchmark can afford, "50%" is a sentence about two runs and the reader has to be able to see that.

func (Rate) OK 

func (r Rate) OK() bool

OK reports whether the rate rests on any graded run at all.

func (Rate) StdErr 

func (r Rate) StdErr() float64

StdErr is the standard error of the proportion, sqrt(p(1-p)/n) -- the cheap honest answer to "how much of this is noise?". At n=2, p=0.5 it is 0.35, which is the point.

func (Rate) String 

func (r Rate) String() string

String renders the rate with its counts: "0.50 (1/2)".

func (Rate) Value 

func (r Rate) Value() float64

Value is Passed/Graded, or 0 when nothing was graded. Check OK first: a zero from an empty cell and a genuine 0% are not the same claim.

type Result 

type Result struct {
	Pass    bool
	Details []string // one human-readable line per check
	Metrics Metrics
}

Result is one graded run: the verdict, the per-check trace behind it, and the measurements the report aggregates.

type Results 

type Results struct {
	Schema      int       `json:"schema"`
	GeneratedAt time.Time `json:"generatedAt"`
	// Root is the run tree the set was collected from, for provenance only;
	// nothing reads back through it.
	Root string `json:"root,omitempty"`
	// Control is the condition name the uplift metric subtracts -- the arm with
	// the vanilla profile. Empty means there is none, and ControlNote says why.
	Control     string `json:"control,omitempty"`
	ControlNote string `json:"controlNote,omitempty"`
	// Baseline and Candidate name the two version labels being compared, when
	// the run tree was produced by a version comparison.
	Baseline  string      `json:"baseline,omitempty"`
	Candidate string      `json:"candidate,omitempty"`
	Runs      []RunResult `json:"runs"`
}

Results is the whole graded results set -- every cell of the scenario x condition x version x run matrix that was captured.

func Collect 

func Collect(ctx context.Context, root string, opt CollectOptions) (Results, error)

Collect walks a run tree, grades what needs grading, and assembles the results set. Runs already carrying a grade.json are reused as-is unless Regrade is set, so re-reporting is free and a fixed grader is one flag away.

Grading a run writes its grade.json back into the run dir. A run tree that cannot be written to still reports -- the cache is a convenience, not the record.

func NewResults 

func NewResults(root string, runs []RunResult) Results

NewResults assembles a results set and derives its control arm.

func ReadResults 

func ReadResults(path string) (Results, error)

ReadResults reads an exported results set back. A schema from the future is an error rather than a partial parse: a reader that silently drops fields it does not know would report a confidently wrong number.

func (Results) AggregateUplifts 

func (r Results) AggregateUplifts(version string) []Uplift

AggregateUplifts returns one row per condition, pooled over every scenario.

func (Results) Cell 

func (r Results) Cell(scenario, condition, version string) (CellStats, bool)

Cell summarizes one cell, reporting whether it holds any run at all.

func (Results) Cells 

func (r Results) Cells() []CellStats

Cells summarizes every populated cell, in report order.

func (Results) Conditions 

func (r Results) Conditions() []string

Conditions lists the condition names present, ordered the way the arms are meant to be read: the control first, then increasing amounts of Seamless.

func (Results) ControlDrift 

func (r Results) ControlDrift(baseline, candidate string) (Rate, Rate, bool)

ControlDrift is the control arm's own pass-rate at each version -- the invariant that calibrates base-model and environment noise across a version comparison. It reports false when the set has no control arm.

func (Results) MinGradedPerCell 

func (r Results) MinGradedPerCell() int

MinGradedPerCell is the smallest graded-run count over the populated cells -- the n that qualifies every number in the report. Zero when nothing is graded.

func (Results) RateFor 

func (r Results) RateFor(scenario, condition, version string) Rate

RateFor is the pass-rate over a selector, pooling every graded run it covers. An empty scenario pools across scenarios, which is what the aggregate row means: scenarios are weighted by their graded run count, so a cell that lost runs to crashes carries proportionally less of the aggregate.

func (Results) RunCount 

func (r Results) RunCount(scenario, condition, version string) int

RunCount counts every run matching a selector, graded or not.

func (Results) ScenarioUplifts 

func (r Results) ScenarioUplifts(version string) []Uplift

ScenarioUplifts returns one row per populated scenario x condition cell at a version.

func (Results) Scenarios 

func (r Results) Scenarios() []string

Scenarios lists the scenario names present, sorted.

func (Results) Totals 

func (r Results) Totals() (total, graded, failedToRun, ungradeable int)

Totals counts the whole set by status, for the header line that tells a reader how much of the matrix actually produced evidence.

func (Results) UpliftFor 

func (r Results) UpliftFor(scenario, condition, version string) Uplift

UpliftFor computes one condition's uplift over the control. Scenario "" is the aggregate.

func (Results) VersionDeltas 

func (r Results) VersionDeltas(baseline, candidate string) []VersionDelta

VersionDeltas compares every condition's uplift between two versions, per scenario and aggregate (Scenario ""). A negative Value is a regression: Seamless helped less on the candidate than it did on the baseline.

Read these next to ControlDrift. The control arm has no Seamless in it, so any movement there is the base model or the run environment, not this repo -- which is what tells "Seamless got worse" apart from "the model had a bad day".

func (Results) Versions 

func (r Results) Versions() []string

Versions lists the version labels present, baseline first when the set knows which is which, else sorted.

type RunArtifacts 

type RunArtifacts struct {
	Scenario   string
	Condition  Condition
	Dir        string // the run's artifact directory
	RepoDir    string // preserved copy of the arm's demo-repo working tree after the run
	RepoDiff   string // unified git diff against the pre-run snapshot
	DataDir    string // preserved copy of the arm's Seamless data dir; "" on vanilla arms
	Transcript string // path to the copied agent transcript (.jsonl); "" if none was produced
	// Steps holds the EARLIER sessions' preserved artifacts of a multi-step
	// run, oldest first; the final session's transcript and tree are the
	// top-level fields above. Empty for a single-session run.
	Steps []StepArtifacts
}

RunArtifacts is what the headless runner captures from one completed run and hands to the grader. Every path points inside the run's preserved artifact directory (see RunDir), not at the live arm, so grading is fully decoupled from running: a run dir can be graded later, on another machine, or synthesized by a test.

type RunRecord 

type RunRecord struct {
	Scenario  string    `json:"scenario"`
	Condition Condition `json:"condition"`
	Run       int       `json:"run"` // 1-based index within the scenario x condition cell
	Version   string    `json:"version"`
	Model     string    `json:"model,omitempty"`
	Prompt    string    `json:"prompt,omitempty"`
	StartedAt time.Time `json:"startedAt"`
	EndedAt   time.Time `json:"endedAt"`
	ExitCode  int       `json:"exitCode"`
	// Error is non-empty when the run itself failed (agent crash, timeout,
	// harness error) as opposed to the agent simply doing the wrong thing.
	// A failed run is not a graded failure; the report counts it separately.
	Error   string  `json:"error,omitempty"`
	Metrics Metrics `json:"metrics"`
	// Concurrent records that this run shared the machine with the other
	// conditions of its scenario (`run --parallel-conditions`). Turns, tokens
	// and cost are unaffected by that; WALL-CLOCK IS. A concurrent run's
	// durationMs carries contention from two other agent sessions and their
	// daemons, so it is not comparable with a serial run's -- including across
	// the two halves of a version comparison, where a mode difference would
	// read as a timing regression. Absent on serial runs, whose manifests are
	// unchanged.
	Concurrent bool `json:"concurrent,omitempty"`
	// Steps is a multi-step run's per-session breakdown, in run order, holding
	// every session that was ATTEMPTED (a step the run never reached is not
	// recorded). Prompt is empty on such a run -- the prompts live on the
	// steps -- and the top-level Metrics is the runner-half sum over them
	// (SumRunnerMetrics), so every existing reader of run.json keeps working.
	// Absent on single-step runs, whose manifests are unchanged.
	Steps []StepRecord `json:"steps,omitempty"`
}

RunRecord is the manifest one completed run leaves in its artifact directory. It carries everything the report needs to place the run in the matrix (scenario x condition x version x run index) without re-deriving it from directory names.

func ReadRunRecord 

func ReadRunRecord(dir string) (RunRecord, error)

ReadRunRecord reads the manifest from a run directory.

type RunResult 

type RunResult struct {
	Record RunRecord `json:"record"`
	// Dir is the run directory, slash-separated and relative to the results
	// root, so an exported results set survives being moved or copied.
	Dir   string `json:"dir,omitempty"`
	Grade Grade  `json:"grade"`
}

RunResult is one run in a results set: what the runner recorded and what the grader made of it.

func (RunResult) Metrics 

func (r RunResult) Metrics() Metrics

Metrics returns the run's full measurement set: the runner's half from the manifest merged with the grader's half from the verdict.

type RunStatus 

type RunStatus string

RunStatus classifies one run for aggregation. 

const (
	// StatusGraded is a run whose verdict is real evidence about the agent.
	StatusGraded RunStatus = "graded"
	// StatusFailedToRun is a run that never produced a verdict to begin with:
	// the agent crashed, timed out, or the capture failed. Counted apart from
	// failures because it says nothing about the agent.
	StatusFailedToRun RunStatus = "failed_to_run"
	// StatusUngradeable is a run that finished but cannot be graded -- missing
	// artifacts, an unreadable database, a manifest naming a scenario this
	// build does not know. Also counted apart: no evidence is not bad evidence.
	StatusUngradeable RunStatus = "ungradeable"
)

type Scenario 

type Scenario struct {
	// Name identifies the scenario in condition matrices, trial tags, and
	// reports.
	Name string
	// Prompt is the user prompt the headless runner feeds the agent, the same
	// on every arm. It is sugar for a single-session scenario; a multi-session
	// scenario sets Steps instead, and setting both is a table error
	// (selectScenarios and bench_test refuse it).
	Prompt string
	// Steps is the ordered agent-session list for a multi-session scenario.
	// Leave nil and set Prompt for the common single-session case.
	Steps []Step
	// Seed builds the scenario's fixture state (memories, plan tasks,
	// findings, trials) via demokit.
	Seed SeedFunc
	// Grader scores a captured run; nil until the grader step of
	// plan:seambench lands.
	Grader Grader
	// RequiresRecall marks scenarios whose signal depends on the mid-session
	// UserPromptSubmit <seam-recall> injection. Headless `claude -p` fires
	// SessionStart but not UserPromptSubmit, so these cannot run as plain -p
	// takes; the runner validates the recall mechanism at the hook-API level
	// instead (memory headless-cc-p-skips-userpromptsubmit-hook). Incompatible
	// with Steps: the recall path is a component check, not a session sequence.
	RequiresRecall bool
}

Scenario is one benchmark scenario: a seeded starting state, the prompt the agent gets, and how the outcome is graded.

func ScenarioByName 

func ScenarioByName(name string) (Scenario, bool)

ScenarioByName returns the named scenario, reporting whether it exists.

func Scenarios 

func Scenarios() []Scenario

Scenarios returns the benchmark scenario table.

func (Scenario) Sessions 

func (sc Scenario) Sessions() []Step

Sessions normalizes the scenario to its ordered session list: Prompt is sugar for a single anonymous step.

type SeedFunc 

type SeedFunc func(s *demokit.Seeder, repoPath string) error

SeedFunc builds a scenario's fixture state inside a fresh throwaway Seamless data dir. The runner hands it a demokit seeder already opened on that dir, plus the arm's demo-repo path to map to the scenario's project so sessions starting there bind to it ("" skips the mapping). It writes only to the data dir and DB, never into the repo working tree (memory scene-demo-repo-must-be-seamless-free), and must never be pointed at a live instance -- demokit.New's contract.

type Step 

type Step struct {
	// Name labels the step in artifacts and logs ("investigate", "fix"); the
	// runner falls back to a positional label when empty.
	Name string
	// Prompt is what this step's agent session is asked.
	Prompt string
	// Evidence maps repo-relative paths to contents the runner materializes
	// into the working tree before this step and removes after it. Evidence is
	// scenario WORLD-STATE (an incident log the agent was pointed at), never
	// Seamless scaffolding, so it does not breach the seeds-write-only-to-the-
	// data-dir rule (memory scene-demo-repo-must-be-seamless-free). Paths must
	// be repo-local and must not collide with a file the fixture ships; the
	// runner removes the files before any diff or capture, so evidence never
	// appears in a graded tree.
	Evidence map[string]string
	// FreshRepo resets the working tree to the arm snapshot before this step,
	// so nothing the previous step's agent left in the tree -- notes files
	// included -- carries over. The Seamless data dir is deliberately NOT
	// reset: persistence across sessions is the thing being measured, and on a
	// vanilla arm nothing persists, which is exactly the control.
	FreshRepo bool
}

Step is one agent session within a scenario, run in order. Each step is its own headless invocation, so SessionStart (and the briefing on a Seamless-ful arm) fires per step -- which is exactly what a handoff scenario measures: what one session recorded is all a later session can inherit.

type StepArtifacts 

type StepArtifacts struct {
	Name       string
	RepoDiff   string // that session's diff against the arm snapshot
	Transcript string // path to that session's transcript; "" if none was produced
}

StepArtifacts is one non-final session's preserved evidence in a multi-step run. The final tree is what gets graded; these exist so the judge and a debugging human can see what each earlier session did.

type StepRecord 

type StepRecord struct {
	Name      string    `json:"name"`
	Prompt    string    `json:"prompt"`
	SessionID string    `json:"sessionId,omitempty"`
	StartedAt time.Time `json:"startedAt"`
	EndedAt   time.Time `json:"endedAt"`
	ExitCode  int       `json:"exitCode"`
	Error     string    `json:"error,omitempty"`
	// Metrics is the runner-owned half for this session alone; the grader half
	// spans the whole run (one preserved event log covers every session).
	Metrics Metrics `json:"metrics"`
}

StepRecord is one agent session's slice of a multi-step run's manifest.

type Uplift 

type Uplift struct {
	Scenario  string `json:"scenario,omitempty"`
	Condition string `json:"condition"`
	Version   string `json:"version"`
	// Control names the arm subtracted. HasControl false means the results set
	// has no vanilla arm, so Value is not an uplift at all and Rate is the only
	// honest thing to report.
	Control     string `json:"control,omitempty"`
	HasControl  bool   `json:"hasControl"`
	Rate        Rate   `json:"rate"`
	ControlRate Rate   `json:"controlRate"`
	// Value is Rate - ControlRate, valid only when HasControl and both rates
	// are OK.
	Value float64 `json:"value"`
}

Uplift is the primary metric: how much better a condition did than the control, at one version. Scenario "" is the aggregate over every scenario.

func (Uplift) OK 

func (u Uplift) OK() bool

OK reports whether the uplift figure means anything: a control exists and both arms have at least one graded run.

type VersionDelta 

type VersionDelta struct {
	Scenario  string  `json:"scenario,omitempty"`
	Condition string  `json:"condition"`
	Baseline  Uplift  `json:"baseline"`
	Candidate Uplift  `json:"candidate"`
	Value     float64 `json:"value"`
	// OK is false when either side is missing; Note says which.
	OK   bool   `json:"ok"`
	Note string `json:"note,omitempty"`
}

VersionDelta compares one condition's uplift across two versions. A negative Value is a regression: Seamless helped less than it used to.

type VersionPair 

type VersionPair struct {
	Schema    int    `json:"schema"`
	Baseline  string `json:"baseline"`
	Candidate string `json:"candidate"`
}

VersionPair is the baseline/candidate record a version-comparison run leaves at the root of its run tree.

func ReadVersionPair 

func ReadVersionPair(root string) (VersionPair, bool, error)

ReadVersionPair reads a run tree's baseline/candidate record, reporting whether there was one.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.