Documentation
¶
Overview ¶
Package testquality is the Go TEST-QUALITY ratchet: the Go-side twin of internal/pythongate, pointed at tests that pass whether or not the code under test works.
Why fak needs it ¶
fak's known failure family is a green gate that measured nothing: a suite hidden behind `//go:build ignore`, a `go test -overlay` run reporting a false `ok`, a green test that never validated the header comment it was named for. A test that asserts nothing is that same failure with no build tag to give it away — it executes code and then reports success unconditionally. fak already scores test quality (internal/brittleness, internal/mutationefficacy, internal/mutationbudget) but had no LANDING-TIME floor on it, and the one landing-time floor of this shape it does run (pythongate's NEW_PYTHON_TOOL) covers Python only.
The ratchet contract, in four clauses ¶
Every clause here is load-bearing; dropping any one turns the gate into either noise or a silent pass.
KEYS STABLE UNDER EDITING. A finding's identity is (code, file, test function) — never the line. Reformat the file, insert an assertion above the finding, move the function: the key survives. A line-keyed baseline goes stale on any edit above a finding and re-reports work nobody touched.
COUNTS, NOT LINES. The baseline stores how MANY findings of each key the tree already carries. Two findings of the same code in the same function have no stable distinguishing identity — the second one is only "the second one", and inserting a line above it would renumber it. Counting means fixing one of two still tightens the floor on regeneration, and adding a third is still caught.
HARD-FAIL ON AN UNPARSEABLE BASELINE ROW. Blank lines and `#` comments are skipped; everything else that does not parse is an ERROR naming the line number, never a skipped line. A lenient parser that dropped a malformed row would read that key's floor as ZERO, so the ratchet's own bug would present as a fresh finding in somebody else's diff — and, on the other side of the same coin, a typo'd key would permanently absorb a real finding.
ONLY EVER CLAIM "NOT GROWING". The verdict is about the DELTA. This package never says a tree is clean, because it cannot: a finding is a CANDIDATE, and some candidates are correct as written (see the deliberate rows in baseline.txt). A tool that refused on any candidate would be wrong most of the time and would simply be switched off — and a switched-off checker reports zero findings, which is byte-identical to a clean tree.
What it reports ¶
TESTQ_NO_ASSERTION a TestXxx with no reachable failure call at all: no
t.Error/t.Fatal, no t.Skip, and the *testing.T is
never handed to a helper to assert on its behalf.
TESTQ_SELF_COMPARISON an assertion comparing a value to itself (`got ==
got`, `reflect.DeepEqual(want, want)`) — true by
construction, so it cannot fail.
TESTQ_UNCHECKED_ERR an error captured and then never inspected before it
is overwritten or the function ends: the failure that
call can return cannot fail the test.
TESTQ_UNREAD_EXPECTATION a table-test row field named like an expectation
(want…/expect…/golden…) that no code in the test ever
reads — the table documents an assertion it does not
make.
Why it under-reports on purpose ¶
Every judgement call is resolved toward silence, because a checker with false positives gets disabled and then reports nothing at all — the same output as a clean tree. The costs, named so they are not mistaken for coverage:
- Errors are recognised by IDENTIFIER NAME (case-insensitive `err`, or the exact name `e`), not by type. A test that calls its error `problem` is invisible. Fixing that needs go/types and a full package load.
- Handing the *testing.T to any function counts as delegating the assertion, so `wantRefusal(t, err, "…")` is trusted rather than followed. A helper that asserts nothing therefore launders every caller.
- An expectation field is read if the row variable's field is selected ANYWHERE in the test, and the whole rule stands down if the row is ever passed somewhere as a value.
- `!=` self-comparison is not reported in a function that mentions NaN, which is the one place `x != x` is a real check.
Scope ¶
The corpus is the git-TRACKED `*_test.go` set, not a filesystem walk. On fak's shared many-session checkout an untracked peer scratch file is not part of anybody's commit, and a gate whose verdict is a function of peer WIP is a gate that refuses work nobody did.
Regenerating the baseline ¶
fak test-quality --write-baseline
Regeneration only ever records TODAY's tree. Because a fixed finding is simply absent from the new scan, the regenerated floor is at or below the old one for every key that was fixed — the ratchet tightens, and `fak test-quality` names each loose key so the tightening is visible rather than silent.
Index ¶
- Constants
- Variables
- func CountsByCode(findings []Finding) map[string]int
- func FormatBaseline(findings []Finding) []byte
- func FormatBaselineWithNotes(findings []Finding, prior []byte) []byte
- func IsSkipped(rel string) bool
- func TrackedTestFiles(root string) ([]string, error)
- func TrailingNotes(src []byte) []byte
- type Baseline
- type Finding
Constants ¶
const ( // CodeNoAssertion — a TestXxx with no reachable failure call: no // t.Error/t.Fatal, no t.Skip, and the *testing.T is never handed to a helper. CodeNoAssertion = "TESTQ_NO_ASSERTION" // CodeSelfComparison — an assertion comparing a value to itself, which is true // by construction and therefore cannot fail. CodeSelfComparison = "TESTQ_SELF_COMPARISON" // CodeUncheckedErr — an error captured and never inspected before it is // overwritten or the function ends. CodeUncheckedErr = "TESTQ_UNCHECKED_ERR" // CodeUnreadExpectation — a table-test row field named like an expectation that // no code in the test ever reads. CodeUnreadExpectation = "TESTQ_UNREAD_EXPECTATION" )
The finding codes. They are constants because the baseline file is KEYED on them: a typo in one place would silently open a hole in the ratchet (the mistyped key's floor reads as zero, so every finding of that shape reports as new — or, if the typo is in the baseline, the real key's floor vanishes).
const BaselineFile = "internal/testquality/baseline.txt"
BaselineFile is the accepted floor's tracked location, repo-relative. It lives beside the package that reads it so the gate and the tool can never end up pointing at two different files.
Variables ¶
var Codes = []string{CodeNoAssertion, CodeSelfComparison, CodeUncheckedErr, CodeUnreadExpectation}
Codes is the closed set, in report order. The baseline parser rejects any code outside it, so a renamed constant cannot leave orphaned floor rows behind that silently absorb a real finding.
Functions ¶
func CountsByCode ¶
CountsByCode folds findings into a per-code tally, for the whole-tree report.
func FormatBaseline ¶
FormatBaseline renders findings as a baseline file, sorted so regenerating it on an unchanged tree produces byte-identical output. A baseline whose line order depended on a map walk would show up as a diff in every commit that touched it, and a noisy generated file stops being read.
func FormatBaselineWithNotes ¶
FormatBaselineWithNotes is FormatBaseline with prior's trailing note block carried across, so regenerating the floor tightens the counts without erasing the review that justified them. prior may be nil or noteless; then this is exactly FormatBaseline.
func IsSkipped ¶
IsSkipped reports whether a repo-relative path sits under a directory the scan never judges. Exported so the hooks gate filters the tracked tree the same way the tool does, instead of maintaining a second, drifting copy of the rule.
func TrackedTestFiles ¶
TrackedTestFiles lists the git-TRACKED *_test.go paths under root, sorted and slash-separated.
Tracked, not walked. fak's checkout is shared by ~20 concurrent agent sessions, so the working tree carries files that are in nobody's commit; a gate whose verdict is a function of peer WIP refuses work the author did not do. It is the same argument internal/pythongate makes for `git ls-files tools/*.py`.
An empty result is an ERROR, not a clean tree. A scan whose corpus went to zero (wrong root, a git that failed silently) reports zero findings, which is indistinguishable from a tree with none — the false-negative shape this whole package exists to refuse.
func TrailingNotes ¶
TrailingNotes returns the comment block a baseline file carries AFTER its last data row — the human record of which candidates were reviewed and judged deliberate, and why.
It is extracted separately because FormatBaseline rewrites the file from the findings alone, and the header it writes tells its reader to regenerate with `fak test-quality --write-baseline`. A writer that did not carry these lines over would delete the tree's only statement of WHY a row is deliberate, on the exact command the file recommends — and the loss is silent, because the regenerated file still parses and still ratchets. Only the audit trail is gone.
Blank lines inside the block are preserved; trailing blank lines are not. A leading comment block is NOT returned: that is the generated header, and carrying it over would duplicate it on every regeneration.
Types ¶
type Baseline ¶
Baseline is the accepted floor: how many candidates of each kind the tree is already known to carry, keyed by Finding.Key(). A finding is NEW only when its key's running count EXCEEDS the floor.
A ratchet and not a refusal, because a finding is a CANDIDATE: some of them are correct as written (a table whose expectation is asserted through a helper, an error deliberately dropped in a teardown path). A gate that denied on any candidate would be wrong often enough to be switched off, and a switched-off checker reports zero findings — which is byte-identical to a clean tree. "This class is not growing" is the strongest TRUE statement available here.
func LoadBaseline ¶
LoadBaseline reads and parses the tracked floor under root. A MISSING baseline is reported as such (nil, false, nil) rather than as an empty one: an absent floor makes every candidate "new", which is the honest reading but must be said out loud, because otherwise it looks like a sudden regression in the diff of whoever happens to run next. An unparseable baseline is a hard error — see ParseBaseline.
func ParseBaseline ¶
ParseBaseline reads the tab-separated baseline format:
CODE<TAB>path/to/file_test.go<TAB>TestFuncName<TAB>count
Blank lines and `#` comments are skipped. EVERYTHING else that does not parse is an error naming the line number, never a skipped line. A baseline that silently dropped a malformed entry would read as a floor of ZERO for that key, so the ratchet's own bug would present as a fresh finding in somebody else's diff; and a typo'd key that parsed anyway would permanently absorb a real finding. Both directions are silent, which is why neither is tolerated.
type Finding ¶
type Finding struct {
Code string `json:"code"`
File string `json:"file"` // slash-separated, relative to the repo root
Func string `json:"func"`
Line int `json:"line"`
Detail string `json:"detail"`
}
Finding is one CANDIDATE — a test-quality defect the analyzer can see the SHAPE of. Whether it is a real defect is a judgement this package never makes; see the package doc's clause 4.
Func is part of the identity and Line is NOT. Line is carried for the report (so an editor and a terminal both make it clickable) and deliberately excluded from Key: a line-keyed baseline goes stale the moment anything above the finding moves.
func Analyze ¶
Analyze reports the candidates in one test file. name is used for Finding.File and for the parse error; src is the file's bytes.
A parse error is RETURNED, never swallowed. A file this package cannot read is a file it cannot judge, and reporting zero findings for it would be a lie in the shape of a pass — the exact failure mode the whole package exists to catch.
func NewFindings ¶
NewFindings splits findings into those the baseline already accounts for and those it does not, and reports which baseline keys are now LOOSER than the tree (a fix landed, so the floor can be lowered).
findings must already be in a deterministic order — Analyze sorts by line and Scan sorts by path — so which of two same-key findings is called "new" is stable rather than whichever the walk happened to reach first.
func Scan ¶
Scan analyzes every named file (repo-relative, slash-separated) under root and returns the findings in a deterministic order: by path, then by Analyze's within-file order.
A file that cannot be read or parsed aborts the scan with an error. A partial scan reported as a result would understate the tree and, through the ratchet, silently lower the floor on regeneration.
Source Files