affectedtests

Documentation

Overview

Package affectedtests is the pure core of the `fak affected` fast test gate: given the package import graph and the set of CHANGED packages, it computes the exact set of packages whose test outcome could change -- so a developer runs `go test` on only those, turning the full ~minutes `go test ./...` into a seconds-long pre-commit gate WITHOUT dropping coverage on what they changed.

Tier: foundation (1) -- see internal/architest. It is a pure graph primitive: it imports only the standard library, has no I/O, and never touches the request path. The impure shell (cmd/fak/affected.go) gathers the inputs -- `git diff` for the changed files, `go list -json ./...` for the import graph -- and runs `go test` on the result; this package owns only the deterministic selection.

THE CORRECTNESS ARGUMENT. A package P's tests can only behave differently than they did at the base if P itself changed, or if some package P transitively imports changed. Equivalently: the affected set is the changed packages together with all their ANCESTORS in the import DAG (every package with an import path leading to a changed one). Test imports count as import edges -- if P's _test.go imports changed Q, P is affected. So as long as the edge set the shell hands in includes test imports, selecting the ancestor closure can never skip a package whose test could newly fail. It is conservative in exactly one safe direction: a package imported by no test and only by far-away production code is still re-tested (a true edge, not a false one); it never DROPS a package that a sound full run would have caught.

THE LIMIT, STATED. This is an IMPORT-graph closure, not a behavioral one. It assumes a test's outcome depends only on Go packages reachable through the import graph. A test that reads a file at runtime by a path the shell did not map to a package, talks to a network service, or depends on build tags the graph did not expand, can change without its package being selected. That residue is why `make ci` still runs the full `go test ./...` as the authoritative gate -- `fak affected` is the fast INNER loop, not a replacement for the full oracle. Naming the limit here is the honesty fence (docs/standards/net-true-value.md Q3 scope).

Index

• Constants
• func ChangedPackages(fileToPkg map[string]string, files []string) []string
• func ClassifyReruns(initialFailed []string, passedOnRerun map[string]bool) (flaky, stillFailing []string)
• func FailedPackages(output string) []string
• func PassedPackages(output string) []string
• func Select(edges map[string][]string, changed []string) []string
• type Blame
  • func Attribute(failing []string, mineClosure, baselineRed, baselineSeen map[string]bool, ...) []Blame
• type ComparisonArm
• type ComparisonResult
  • func CompareLocal() ComparisonResult
• type Finding
  • func ClassifyRerunFindings(firstRun, reruns []TestEvent) (flaky, stillFailing []Finding)
  • func (f Finding) Qualified() string
• type PackageObservation
• type SelectionAudit
  • func AuditSelection(selected, full TestObservation) SelectionAudit
• type TestEvent
  • func ParseTestEvents(raw string) []TestEvent
• type TestObservation

Constants

const (
	// BlameMine: the red is attributable to the caller's declared change — in the
	// --mine closure (or nothing was declared) and not exonerated by the baseline.
	// The only class that fails the gate.
	BlameMine = "mine"
	// BlamePeerWIP: the failing package is OUTSIDE the closure of the caller's
	// declared files — the red comes from some other working-tree change (a peer's
	// uncommitted WIP), not from the caller's diff.
	BlamePeerWIP = "peer-wip"
	// BlamePeerPreexisting: the package is red at a CLEAN checkout of the base ref —
	// the red pre-dates the caller's diff entirely.
	BlamePeerPreexisting = "peer-preexisting"
)

The closed blame vocabulary (#2138). String constants in the same shape as the leaseref liveness classes; they are the JSON contract a calling loop routes on.

const FlakyPassedOnRetry = "FLAKY_PASSED_ON_RETRY"

FlakyPassedOnRetry is the closed verdict a run earns when EVERY initially-failing package passed on a same-tree rerun — non-deterministic, not a deterministic regression from the caller's diff. Named alongside the blame classes as the JSON contract a calling loop routes on; the `fak affected` shell promotes the run's verdict to this string and drops the exit to 0 when stillFailing is empty.

const StillFailing = "STILL_FAILING"

StillFailing is the per-test verdict for a test that failed the first run and never produced a positive PASS on any same-tree rerun — the fail-closed default (a red without green evidence is a real red, not a flake). It is the per-test counterpart of the package staying in ClassifyReruns' stillFailing set.

Functions

func ChangedPackages

func ChangedPackages(fileToPkg map[string]string, files []string) []string

ChangedPackages maps a set of changed FILE paths to the set of packages they belong to, using a precomputed FILE->import-path index (fileToPkg, keyed by each package's actual source/embed file at its repo-relative slash path -- the shell builds it from `go list`'s GoFiles / TestGoFiles / XTestGoFiles / EmbedFiles / Cgo / ignored-by-build lists). Mapping by real source-file membership rather than by directory is what keeps the selection both precise and correct at the module root: a top-level Makefile, a README, or a doc inside a package directory is NOT one of any package's source files, so it maps to nothing -- a docs/build-only change selects an empty set and skips the suite, and a non-source file never spuriously drags in the root package.

The result is sorted and de-duplicated. Pure and deterministic.

func ClassifyReruns

func ClassifyReruns(initialFailed []string, passedOnRerun map[string]bool) (flaky, stillFailing []string)

ClassifyReruns splits the initially-failing packages into the FLAKY ones (failed the first run, then produced a passing verdict on a later same-tree rerun) and the ones STILL FAILING after every rerun. passedOnRerun is the set of packages that produced a positive `ok` verdict in some rerun round; a package absent from it stays in stillFailing — fail-closed, because flakiness needs positive green evidence and never the mere absence of a repeated FAIL. Both slices are sorted and deduplicated. The union of the two is exactly the deduplicated input, so the caller can trust that an empty stillFailing means every red was exonerated as flaky. Pure and deterministic.

func FailedPackages

func FailedPackages(output string) []string

FailedPackages parses the per-package result lines of plain `go test` output and returns the failing import paths, sorted and deduplicated. It reads only the package-level verdict lines go test always emits —

FAIL<tab>example.com/pkg<tab>0.42s
FAIL<tab>example.com/pkg [build failed]

— matched by the "FAIL\t" COLUMN-ZERO prefix go itself prints, so test-level noise ("--- FAIL: TestX"), the bare trailing "FAIL" summary line, and INDENTED test-log lines that happen to start with FAIL are all excluded. (A test that itself prints a forged "FAIL\tpkg" at column zero to stdout can still spoof a row — un-forgeable attribution would need `go test -json`; named residual, not covered.) An output with no such lines yields an empty slice: the caller must treat "red run, nothing parsed" as unattributable and keep the red exit, never guess. Pure and deterministic.

func PassedPackages

func PassedPackages(output string) []string

PassedPackages is FailedPackages' dual for the "ok \texample.com/pkg\t0.01s" (or "(cached)") verdict lines. The union of the two is the set of packages a run actually PRODUCED A VERDICT for — the baseline-rerun coverage evidence Attribute needs to phrase a mine row honestly.

func Select

func Select(edges map[string][]string, changed []string) []string

Select returns the sorted set of packages whose tests should run given the changed packages: the changed packages themselves PLUS every package that transitively imports a changed package. edges[p] is the list of packages p directly imports (intra-module, and INCLUDING test imports -- the shell is responsible for folding Imports + TestImports + XTestImports). A package that appears only as an import target (never as an edges key) is handled correctly; a changed package with no importers selects just itself.

Pure and deterministic: same inputs -> identical output, always.

Types

type Blame

type Blame struct {
	Package  string `json:"package"`
	Class    string `json:"class"`
	Evidence string `json:"evidence"`
}

Blame is one failing package with its attribution class and the evidence sentence naming the comparison that decided it.

func Attribute

func Attribute(failing []string, mineClosure, baselineRed, baselineSeen map[string]bool, baselineRef string) []Blame

Attribute classifies each failing package. mineClosure is the affected-set closure of the caller's declared --mine files (nil = nothing declared, so every red is closure-attributable to the caller); baselineRed is the set of packages red at a clean checkout of baselineRef and baselineSeen the set the baseline actually PRODUCED A VERDICT for (red or ok) — nil on both means the baseline rerun was unavailable, so no exoneration from that rung, fail-closed. Precedence: a baseline red wins (peer-preexisting), then the closure rung (peer-wip), else mine; the mine evidence distinguishes "green at the baseline" from "the baseline never tested it" (a package new in the diff does not exist at the base ref) so the sentence never claims evidence that was not gathered. The result is sorted by package and deduplicated. Pure and deterministic.

type ComparisonArm

type ComparisonArm struct {
	Name, Kind                                string
	Available, Correct                        bool
	Latency                                   time.Duration
	Packages, Selected, FalseIncludes, Misses int
	CPUSeconds                                float64
	PeakRSSBytes, InputBytes, NetworkBytes    int64
	OperatorSeconds, CostUSD                  float64
	Note                                      string
}

type ComparisonResult

type ComparisonResult struct {
	Workload string
	Arms     []ComparisonArm
}

func CompareLocal

func CompareLocal() ComparisonResult

type Finding

type Finding struct {
	Package string `json:"package"`
	Test    string `json:"test"` // "TestName" or "TestName/subtest" — never just the package
	Verdict string `json:"verdict"`
}

Finding names one individual test/subtest that the reruns classified, carrying the specific Package/Test the package-level ClassifyReruns could only report as a whole poisoned package. Verdict is FlakyPassedOnRetry (failed first, then passed on a same-tree rerun) or StillFailing (never seen green — fail-closed).

func ClassifyRerunFindings

func ClassifyRerunFindings(firstRun, reruns []TestEvent) (flaky, stillFailing []Finding)

ClassifyRerunFindings is the per-test upgrade of ClassifyReruns: it names the individual test/subtest that flaked instead of the whole package. firstRun is the `-json` events of the initial (failing) run; reruns is the concatenated `-json` events of the same-tree rerun round(s). A leaf test that FAILED in firstRun and PASSED in some rerun is FlakyPassedOnRetry; one never seen green stays StillFailing — the SAME fail-closed rule as ClassifyReruns, because a flake needs positive green evidence, never the mere absence of a repeated FAIL.

Only leaf tests are named: when a subtest fails, `go test` also emits a fail for its parent test and the package, but those fail only BECAUSE the subtest did, so an ancestor whose name is a "/"-prefix of another failed name in the same package is dropped — leaving the most specific unit ("TestFoo/case_b"), which is the point of this leaf. Findings are sorted by package then test. Pure.

func (Finding) Qualified

func (f Finding) Qualified() string

Qualified renders the finding as "package.Test" (or just the package when no test is named), the stable key a ledger or ticket dedupes on.

type PackageObservation

type PackageObservation struct {
	Package string `json:"package"`
	Failed  bool   `json:"failed"`
}

PackageObservation records whether a package's test run observed a failure.

type SelectionAudit

type SelectionAudit struct {
	Complete         bool     `json:"complete"`
	Sound            bool     `json:"sound"`
	SelectedFailures []string `json:"selected_failures"`
	FullFailures     []string `json:"full_failures"`
	SelectorMisses   []string `json:"selector_misses"`
}

SelectionAudit compares the failures seen by selected tests with full-suite truth.

func AuditSelection

func AuditSelection(selected, full TestObservation) SelectionAudit

AuditSelection reports full-truth failures that the selected run did not observe. An incomplete selected or truth observation fails closed, even when no miss is visible.

type TestEvent

type TestEvent struct {
	Action  string `json:"Action"`
	Package string `json:"Package"`
	Test    string `json:"Test"`
}

TestEvent is the subset of one `go test -json` event line this fold reads: the Action ("run"|"pass"|"fail"|"output"|…), the Package it happened in, and the Test it names. A package-level event carries an empty Test; a test or subtest event carries "TestName" or "TestName/subtest". Only these three fields drive per-test flake identification, so the rest of the event is ignored.

func ParseTestEvents

func ParseTestEvents(raw string) []TestEvent

ParseTestEvents folds `go test -json` newline-delimited output into events, tolerantly skipping any non-JSON preamble a `go test` run can interleave (build errors, module-download notes, a bare "FAIL" trailer) so a malformed line never discards the whole stream. Pure and deterministic.

type TestObservation

type TestObservation struct {
	Complete bool                 `json:"complete"`
	Packages []PackageObservation `json:"packages"`
}

TestObservation is one selected or full-truth test-run observation.