bench

Documentation

Overview

Package bench implements the evaluator behind `airom bench` (docs/benchmark.md): it scans each corpus entry with the real pipeline, matches the result against hand-written truth labels, and reports the metric set the release gate consumes.

The package computes; it never judges. Every rate is emitted next to its counts, because a rate whose n is hidden is a claim, not a measurement.

Index

• Constants
• func Compare(baseline, current *Report) (failures, remarks []string)
• func NormalizeName(s string) string
• type AttrGrade
• type BandCell
• type Entry
  • func LoadCorpus(root string) ([]Entry, error)
• type Evidence
• type Label
• type LocGrade
• type PRCell
• type RepoResult
  • func Evaluate(repo string, inv *airom.Inventory, truth *Truth) *RepoResult
  • func Run(ctx context.Context, entries []Entry) ([]*RepoResult, airom.ToolInfo, error)
• type Report
  • func Aggregate(airomVersion, rulesVersion, rulesHash string, repos []*RepoResult) *Report
  • func LoadBaseline(path string) (*Report, error)
  • func (r *Report) JSON() ([]byte, error)
  • func (r *Report) Markdown() string
• type ScanSpec
• type Truth
  • func LoadTruth(path string) (*Truth, error)

Constants

const ReportSchemaVersion = 1

ReportSchemaVersion is the bench.json contract (baselines are this format).

const TruthSchemaVersion = 1

TruthSchemaVersion is the truth.yaml contract this evaluator reads.

const VersionUngraded = "*"

VersionUngraded is the Label.Version sentinel that skips version grading.

Functions

func Compare

func Compare(baseline, current *Report) (failures, remarks []string)

Compare applies the gate policy: current against baseline. The returned failures are empty when the gate passes. Improvements are reported as remarks so the caller can prompt a baseline update, never as failures.

func NormalizeName

func NormalizeName(s string) string

NormalizeName folds the spelling differences that never distinguish two real packages: case, and the -/_/. separator churn. Deliberately ASCII-only and length-preserving; a name this cannot fold fails to match, and a missed match is a visible FN, never a corrupted one.

Types

type AttrGrade

type AttrGrade struct {
	Exact    int `json:"exact"`    // label states it, report agrees
	AbsentOK int `json:"absentOk"` // label asserts absence, report is absent
	Missing  int `json:"missing"`  // label states it, report omits it
	Wrong    int `json:"wrong"`    // report contradicts the label
	Ungraded int `json:"ungraded"` // label opted out
}

AttrGrade buckets an attribute over matched components. Wrong is the bucket the gate watches: a wrong claim outranks a missing one everywhere in AIROM, and the benchmark grades by the same rule.

type BandCell

type BandCell struct {
	N       int `json:"n"`
	Correct int `json:"correct"`
}

BandCell is one confidence band's calibration sample.

type Entry

type Entry struct {
	Name  string
	Dir   string // corpus/<name>
	Truth *Truth
}

Entry is one corpus repo: a truth file plus either an extracted tree/ or a snapshot.tar.gz (stdlib-decodable on purpose; the corpus format follows the tool's zero-CGO, zero-extra-dependency discipline).

func LoadCorpus

func LoadCorpus(root string) ([]Entry, error)

LoadCorpus enumerates corpus entries under root/corpus (or root itself if it has no corpus/ subdirectory), validating every truth file up front: a broken label fails the run before any scanning, not mid-report.

type Evidence

type Evidence struct {
	File  string `yaml:"file"`
	Lines []int  `yaml:"lines"` // empty, or [first, last] inclusive
}

Evidence is a location the labeler saw the thing at.

type Label

type Label struct {
	Kind     string `yaml:"kind"`
	Name     string `yaml:"name"`
	Provider string `yaml:"provider"` // "" = provider is not graded for this label
	// Version grades the matched component's version claim. The zero value is
	// an ASSERTION, not an omission: "" means the honest output is an absent
	// version, and a reported one counts as wrong-version (tri-state
	// discipline, §6.4). Labels that do not want version graded at all say
	// version: "*".
	Version  string     `yaml:"version"`
	Scope    string     `yaml:"scope"`  // "" (default presentation) | "test"
	Reason   string     `yaml:"reason"` // forbidden entries: why this is a trap
	Evidence []Evidence `yaml:"evidence"`
}

Label is one expected or forbidden component.

type LocGrade

type LocGrade struct {
	Valid    int `json:"valid"`    // an occurrence lands on labeled evidence
	Invalid  int `json:"invalid"`  // no occurrence does
	Ungraded int `json:"ungraded"` // label carries no evidence
}

LocGrade buckets evidence-location validity over matched components.

type PRCell

type PRCell struct {
	TP int `json:"tp"`
	FP int `json:"fp"`
	FN int `json:"fn"`
}

PRCell is one precision/recall cell: reported, matched, labeled.

type RepoResult

type RepoResult struct {
	Repo string `json:"repo"`

	TP int `json:"tp"` // matched (label found)
	FP int `json:"fp"` // reported, no label
	FN int `json:"fn"` // labeled, not reported

	PerKind map[string]*PRCell `json:"perKind"`
	PerLang map[string]*PRCell `json:"perLang"`

	// Attribute grades over matched components (docs/benchmark.md §4:
	// attributes are graded, not matched on).
	Version  AttrGrade `json:"version"`
	Provider AttrGrade `json:"provider"`
	Location LocGrade  `json:"location"`

	// Traps: forbidden labels that were reported anyway. Each is also an FP;
	// the class is separate because each trap encodes a lesson already
	// learned once.
	TrapViolations []string `json:"trapViolations,omitempty"`

	// Coverage, straight off the assurance block.
	FilesProcessed int64 `json:"filesProcessed"`
	Unknowns       int   `json:"unknowns"`
	FilesTruncated int64 `json:"filesTruncated"`

	// Bands feed the calibration study: per confidence band, how many
	// reported components and how many of them were correct (= matched).
	Bands map[string]*BandCell `json:"bands"`
}

RepoResult is the metric set for one corpus entry. Every rate the report derives from it is printed next to these counts.

func Evaluate

func Evaluate(repo string, inv *airom.Inventory, truth *Truth) *RepoResult

Evaluate grades one scan against one truth. Pure: no I/O, deterministic.

func Run

func Run(ctx context.Context, entries []Entry) ([]*RepoResult, airom.ToolInfo, error)

Run scans every entry and evaluates it, returning the results and the ToolInfo of the ruleset that produced them (every result is stamped with the rules that made it, same as every AIBOM). Scans are offline and fixed-config (no CVE, no EOL, no rule auto-update): the benchmark grades detection, and a metric that moves when OSV.dev does is not measuring the scanner.

type Report

type Report struct {
	SchemaVersion int    `json:"schemaVersion"`
	AiromVersion  string `json:"airomVersion"`
	RulesVersion  string `json:"rulesVersion,omitempty"`
	RulesHash     string `json:"rulesHash,omitempty"`

	Totals *RepoResult   `json:"totals"` // Repo == "TOTAL"
	Repos  []*RepoResult `json:"repos"`
}

Report is the aggregate over every corpus entry plus the per-repo detail.

func Aggregate

func Aggregate(airomVersion, rulesVersion, rulesHash string, repos []*RepoResult) *Report

Aggregate folds per-repo results into one report. Deterministic: repos are sorted, map keys serialize sorted (encoding/json), and nothing samples time.

func LoadBaseline

func LoadBaseline(path string) (*Report, error)

LoadBaseline reads a previously written bench.json.

func (*Report) JSON

func (r *Report) JSON() ([]byte, error)

JSON serializes the report; this is the baseline format the gate reads.

func (*Report) Markdown

func (r *Report) Markdown() string

Markdown renders the human half. Every rate prints beside its counts: a rate whose n is hidden is a claim, not a measurement (docs/benchmark.md §6).

type ScanSpec

type ScanSpec struct {
	Args []string `yaml:"args"`
}

ScanSpec is reserved for per-repo scan configuration. v1 accepts only an empty one: a corpus entry that needs non-default flags is a corpus entry whose numbers are not comparable to the rest, and supporting that starts as a refusal, not a silent knob.

type Truth

type Truth struct {
	SchemaVersion int      `yaml:"schemaVersion"`
	Labeler       string   `yaml:"labeler"`
	Scan          ScanSpec `yaml:"scan"`
	Expected      []Label  `yaml:"expected"`
	Forbidden     []Label  `yaml:"forbidden"`
	Notes         string   `yaml:"notes"`
}

Truth is one corpus entry's ground truth (docs/benchmark.md §4).

func LoadTruth

func LoadTruth(path string) (*Truth, error)

LoadTruth reads and validates one truth.yaml.