benchloop

Documentation

Overview

Package benchloop folds fak's benchmark surfaces into one read-only control loop.

Tier: foundation (1) - see internal/architest. This package imports only same-tier benchmark catalog/query leaves plus stdlib, and stays off the live request path.

Index

• Constants
• func RenderNext(a Action) string
• func RenderStatus(r Report) string
• func RenderWalk(surfaces []Surface) string
• type Action
• type AuthorityStatus
• type CatalogStatus
• type CrossCommit
  • func DefaultCrossCommit(root string) CrossCommit
• type LedgerStatus
• type LineageKey
• type LocalStatus
• type NextTask
• type Options
• type Parts
• type RegistryStatus
• type Report
  • func Load(opts Options) Report
  • func StatusFromParts(p Parts) Report
• type ReuseVerdict
  • func LineageReuse(runs []benchruns.Run, key LineageKey) ReuseVerdict
  • func LineageReuseAcross(runs []benchruns.Run, key LineageKey, cc CrossCommit) ReuseVerdict
• type Surface
  • func Walk() []Surface

Constants

const NoReuseEnv = "FAK_BENCH_NO_REUSE"

NoReuseEnv, set to a truthy value, forces the next benchmark run to execute even when a prior catalog run already covers the same lineage — the force-rerun escape hatch #4600 requires (its explicit non-goal fence). It is the in-lane, operator- and test-settable switch; a cmd-lane `--no-reuse` flag that sets it is a follow-up.

const StatusSchema = "fak.bench-loop.status.v1"

Functions

func RenderNext

func RenderNext(a Action) string

func RenderStatus

func RenderStatus(r Report) string

func RenderWalk

func RenderWalk(surfaces []Surface) string

Types

type Action

type Action struct {
	Kind    string `json:"kind"`
	Command string `json:"command"`
	Detail  string `json:"detail"`
}

type AuthorityStatus

type AuthorityStatus struct {
	Date           string `json:"date"`
	NewerThan      int    `json:"newer_than"`
	TotalRows      int    `json:"total_rows"`
	TotalCollected int    `json:"total_collected"`
}

type CatalogStatus

type CatalogStatus struct {
	Path            string `json:"path"`
	Present         bool   `json:"present"`
	Error           string `json:"error,omitempty"`
	RunCount        int    `json:"run_count"`
	MachineCount    int    `json:"machine_count"`
	LatestRunID     string `json:"latest_run_id,omitempty"`
	LatestMachine   string `json:"latest_machine,omitempty"`
	LatestModel     string `json:"latest_model,omitempty"`
	LatestPrecision string `json:"latest_precision,omitempty"`
	LatestTimestamp string `json:"latest_timestamp,omitempty"`
}

type CrossCommit

type CrossCommit struct {
	// Artifact decodes catalog run r's benchmark_artifact. ok=false means the
	// artifact is unreadable; that run then stays under the commit-exact rule.
	Artifact func(r benchruns.Run) (art benchcli.BenchmarkArtifact, ok bool)
	// ChangedPaths lists the paths changed between the prior run's commit and the
	// current HEAD (git diff --name-only prev..head). ok=false means the window
	// could not be resolved (unknown commit, no git) — fail closed, no reuse.
	ChangedPaths func(prevCommit, headCommit string) (paths []string, ok bool)
	// Next is the identity of the run that would be launched, feeding the
	// model/config drift rungs of DetectInvalidation. Zero-value model+config
	// carries the prior artifact's identity forward: benchloop's launch gate keys
	// on (commit x machine) with model/config wildcards, so the reused artifact
	// IS the config it would re-run.
	Next benchcli.BenchmarkArtifact
}

CrossCommit supplies the ingredients for the finer #5088 freshness predicate: hooks to decode a prior catalog run's benchmark_artifact envelope and to list the repo paths changed between that run's commit and the current HEAD, plus an optional identity for the run that would be launched. With both hooks set, a prior run at a DIFFERENT commit is still reusable when benchcli.DetectInvalidation judges the commit window untouched (no bench code/harness/model/config change). The zero value disables the cross-commit path entirely, leaving #4600's conservative commit-exact rule in force.

func DefaultCrossCommit

func DefaultCrossCommit(root string) CrossCommit

DefaultCrossCommit wires the real evidence hooks for a repo checkout at root: the artifact hook reads the run's recorded path (a report file or a run directory) and decodes the first benchmark_artifact found; the changed-paths hook shells out to git diff --name-only prev..head. Both fail soft to !ok so LineageReuseAcross degrades to the commit-exact rule.

type LedgerStatus

type LedgerStatus struct {
	Path          string `json:"path"`
	Rows          int    `json:"rows"`
	CollectedRows int    `json:"collected_rows"`
	LastDate      string `json:"last_date,omitempty"`
	LastTaskID    string `json:"last_task_id,omitempty"`
	LastOutcome   string `json:"last_outcome,omitempty"`
	LastBox       string `json:"last_box,omitempty"`
}

type LineageKey

type LineageKey struct {
	Commit    string
	Machine   string
	Model     string
	Precision string
}

LineageKey is a benchmark run's reuse identity: the git commit that built the code under test, the machine it ran on, and its model/config. Two runs are interchangeable — one may be skipped in favor of the other — only when these agree. Model and Precision are wildcards when empty, so a caller that knows only (commit × machine) still matches any config recorded for that commit on that box.

type LocalStatus

type LocalStatus struct {
	Capabilities nightrun.Capabilities `json:"capabilities"`
	TaskCount    int                   `json:"task_count"`
	Feasible     int                   `json:"feasible"`
	Blocked      int                   `json:"blocked"`
	Saturated    int                   `json:"saturated"`
	Error        string                `json:"error,omitempty"`
	HasNext      bool                  `json:"has_next"`
	Next         *NextTask             `json:"next,omitempty"`
}

type NextTask

type NextTask struct {
	ID         string  `json:"id"`
	Title      string  `json:"title"`
	Source     string  `json:"source"`
	Value      string  `json:"value"`
	Score      float64 `json:"score"`
	Reason     string  `json:"reason"`
	Run        string  `json:"run"`
	Acceptance string  `json:"acceptance"`
	Manual     bool    `json:"manual,omitempty"`
}

type Options

type Options struct {
	Root        string
	OverlayPath string
	LedgerPath  string
	Now         time.Time
}

type Parts

type Parts struct {
	Root        string
	Now         time.Time
	Commit      string // git HEAD the run under test would be built at (lineage reuse key, #4600)
	Benchmarks  []benchcatalog.Bench
	Catalog     benchruns.Catalog
	CatalogErr  error
	CatalogPath string
	Ledger      []nightrun.CollectRow
	LedgerPath  string
	Caps        nightrun.Capabilities
	Tasks       []nightrun.Task
	TaskErr     error
	Authority   nightrun.LedgerGapReport
	Cross       CrossCommit // #5088 cross-commit reuse evidence hooks; zero value = commit-exact rule only
}

type RegistryStatus

type RegistryStatus struct {
	Benchmarks int `json:"benchmarks"`
	Offline    int `json:"offline"`
	Weights    int `json:"weights"`
	Dataset    int `json:"dataset"`
	Serving    int `json:"serving"`
	E2E        int `json:"e2e"`
	Manual     int `json:"manual"`
}

type Report

type Report struct {
	Schema      string          `json:"schema"`
	GeneratedAt string          `json:"generated_at"`
	Root        string          `json:"root"`
	Registry    RegistryStatus  `json:"registry"`
	Catalog     CatalogStatus   `json:"catalog"`
	Ledger      LedgerStatus    `json:"ledger"`
	Local       LocalStatus     `json:"local"`
	Authority   AuthorityStatus `json:"authority"`
	Reuse       ReuseVerdict    `json:"reuse"`
	NextAction  Action          `json:"next_action"`
	Walk        []Surface       `json:"walk"`
}

func Load

func Load(opts Options) Report

func StatusFromParts

func StatusFromParts(p Parts) Report

type ReuseVerdict

type ReuseVerdict struct {
	Reuse  bool   `json:"reuse"`
	RunID  string `json:"run_id,omitempty"`
	Path   string `json:"path,omitempty"`
	Commit string `json:"commit,omitempty"`
	Reason string `json:"reason,omitempty"`
}

ReuseVerdict is the launch-gate verdict at benchloop.chooseAction: whether the next run can be skipped because a prior catalog run already covers this lineage.

func LineageReuse

func LineageReuse(runs []benchruns.Run, key LineageKey) ReuseVerdict

LineageReuse decides whether any prior run in the catalog makes the next run at key redundant. It is the reuse freshness predicate #4600 asks for: a prior run is reusable only when it was built at the SAME commit (a code/harness/config change moves HEAD, so a commit-exact match is the conservative realization of benchcli.DetectInvalidation's code/model/config-drift rule — a drifted prior run simply fails to match and is re-run) AND ran on the same machine AND, when the caller pins them, the same model/precision. The most recent covering run wins, so the reused artifact is the freshest one for that lineage. An empty key.Commit is a hard no-reuse: without a current commit there is no lineage to match.

func LineageReuseAcross

func LineageReuseAcross(runs []benchruns.Run, key LineageKey, cc CrossCommit) ReuseVerdict

LineageReuseAcross is LineageReuse plus the #5088 cross-commit gate. The exact #4600 predicate runs first and still wins when it finds a same-commit covering run. Only when it does not — and both hooks are live — is the freshest prior run on the other lineage axes (machine/model/precision) re-judged through benchcli.DetectInvalidation over the actual changed paths of prevCommit..HEAD: reuse iff not IsInvalid. Every failure to resolve evidence (no artifact, no diff window) falls back to the conservative no-reuse verdict.

type Surface

type Surface struct {
	Name    string `json:"name"`
	Command string `json:"command"`
	Use     string `json:"use"`
}

func Walk

func Walk() []Surface