evalharness

Documentation

Overview

Package evalharness provides helpers shared by per-language evaluation test suites. Each analyzer package (rustanalyzer, tsanalyzer) has its own eval_test.go that drives the built diffguard binary against a tree of seeded fixtures and diff-compares emitted findings to an expected.json file next to the fixture.

The harness:

• Builds the diffguard binary once per test run (sync.Once inside BuildBinary) to keep the eval suites under 30s wall-clock when the full language set is exercised.
• Copies each fixture into a temp dir before running so fixtures stay pristine regardless of what any analyzer writes (mutation tests swap files in place, so this matters).
• Runs the binary with stable flags (--output json, fixed --mutation-sample-rate, etc.) and returns a decoded report.Report.
• Exposes a semantic equality helper: compares sections by (name, severity) and finding sets by (file, function, severity, operator). Exact counts / percentages / order-within-group are not asserted because sampling and hashmap iteration can shuffle them without changing correctness.

Index

• func AssertMatches(t *testing.T, got report.Report, want Expectation)
• func CopyFixture(t *testing.T, srcDir string) string
• func RepoRoot(t *testing.T) string
• func RunBinary(t *testing.T, binary, repo string, extraArgs []string) report.Report
• type BinaryBuilder
  • func (b *BinaryBuilder) GetBinary(t *testing.T, repoRoot string) string
• type BuildError
  • func (e *BuildError) Error() string
• type Expectation
  • func LoadExpectation(t *testing.T, fixtureDir string) (Expectation, bool)
• type FindingExpectation
• type SectionExpectation

Functions

func AssertMatches

func AssertMatches(t *testing.T, got report.Report, want Expectation)

AssertMatches compares a report against an Expectation and fails the test with human-readable diagnostics on any mismatch. Assertions are semantic: section name (stripped of language suffix), severity, and finding identity — not line-exact counts or percentages.

func CopyFixture

func CopyFixture(t *testing.T, srcDir string) string

CopyFixture mirrors srcDir to a fresh temp dir and returns the path. The copy is rooted at t.TempDir so Go's test harness cleans it up. Directories are preserved but none of the fixture metadata (mode, mtime) is — eval tests don't care.

func RepoRoot

func RepoRoot(t *testing.T) string

RepoRoot walks upward from cwd until it finds go.mod, returning that directory. Eval tests live several packages deep; using this avoids hard-coding relative paths that break when go test is invoked from different working directories (IDE vs. CLI vs. CI).

func RunBinary

func RunBinary(t *testing.T, binary, repo string, extraArgs []string) report.Report

RunBinary runs the diffguard binary against a repo dir with the provided extra flags and returns the decoded JSON report. The harness always sets --output json, --fail-on none (so exit codes don't kill the test), and passes the repo path as the final positional arg.

Types

type BinaryBuilder

type BinaryBuilder struct {
	// contains filtered or unexported fields
}

BinaryBuilder caches the built diffguard binary across tests within a package. Call GetBinary(t) from each test — the first call builds, the rest return the cached path. Using package-level state keeps the cost of running 6+ eval tests in a package to a single build.

func (*BinaryBuilder) GetBinary

func (b *BinaryBuilder) GetBinary(t *testing.T, repoRoot string) string

GetBinary returns the path to a compiled diffguard binary, building it on the first call. Subsequent calls return the same path. The binary lives in os.TempDir; we don't clean it up because keeping the cache warm across tests is worth the few MB.

type BuildError

type BuildError struct {
	Output string
	Err    error
}

BuildError wraps the build command's exit status with the captured combined output so test failures show why the build failed.

func (*BuildError) Error

func (e *BuildError) Error() string

type Expectation

type Expectation struct {
	// WorstSeverity, if non-empty, pins the overall Report.WorstSeverity.
	WorstSeverity report.Severity `json:"worst_severity,omitempty"`
	// Sections pins per-section expectations, keyed by section name
	// (without a language suffix — the harness strips that before
	// matching). If omitted the section is not checked.
	Sections []SectionExpectation `json:"sections,omitempty"`
}

Expectation is the shape of an expected.json file next to each fixture. It captures just the facts worth pinning — the presence/severity of sections and whether each analyzer surfaced any Finding for a given (file, function) key. Fields not listed here are intentionally not asserted on; eval assertions are about "did the right thing get flagged", not "did the output bytes match exactly".

func LoadExpectation

func LoadExpectation(t *testing.T, fixtureDir string) (Expectation, bool)

LoadExpectation reads expected.json from a fixture directory. Returns (Expectation{}, false) if the file doesn't exist.

type FindingExpectation

type FindingExpectation struct {
	File     string          `json:"file,omitempty"`
	Function string          `json:"function,omitempty"`
	Severity report.Severity `json:"severity,omitempty"`
	Operator string          `json:"operator,omitempty"`
}

FindingExpectation is the subset of report.Finding fields used for semantic matching. Unset fields are ignored.

type SectionExpectation

type SectionExpectation struct {
	// Name is the metric prefix without a language suffix, e.g.
	// "Cognitive Complexity" or "Mutation Testing".
	Name string `json:"name"`
	// Severity, if non-empty, pins Section.Severity.
	Severity report.Severity `json:"severity,omitempty"`
	// MustHaveFindings, if non-empty, requires a Finding matching each
	// entry. The harness matches by the fields that are present in the
	// expectation (non-zero values). An expectation with just File set
	// passes if any finding mentions that file, for example.
	MustHaveFindings []FindingExpectation `json:"must_have_findings,omitempty"`
	// MustNotHaveFindings, if true, requires len(Findings)==0 for the
	// section.
	MustNotHaveFindings bool `json:"must_not_have_findings,omitempty"`
}

SectionExpectation pins a single Section's minimum expectations.