go-flake-hunter

Reproduce the flake in 30 seconds instead of 30 minutes — and leave a permanent regression test in the repo.
257 tests · CI on linux/macos/windows + a nightly sweep · zero dependencies ·
three real upstream flake cases reduced to witnesses (evidence)
Finds flaky Go-test failures caused by order, parallelism, repetition and
environment, shrinks them to the smallest witness (polluter → victim) and
generates a regression test you can commit.
It does not detect flaky tests — that layer is crowded (BuildPulse, Trunk,
gotestsum --rerun-fails). This tool does reproduction and
permanence: it turns a failing behavior into a test file that never gets lost again.
go test -json → hypothesis order → ddmin → capsule → regression test
repeat smallest
order polluter→victim
parallel
env
Why not just -count=100?
| Approach |
What it gives you |
What is left for you |
go test -count=100 / -shuffle=on |
a red log, eventually |
explain the failure, keep the flake |
gotestsum --rerun-fails |
a rerun of failed tests |
the flake is still in the repo |
stress, -parallel loops |
a higher reproduction probability |
still no minimal witness |
| go-flake-hunter |
polluter→victim chain, shrunk with ddmin, written as a build-tagged regression test |
the fix |
Detection is not the product; permanence is. A tracker tells you a test is
flaky. go-flake-hunter hands you a test file that stays red until the cause is gone.

Changing the order requires a temporary witness file, because in Go
go test -run '^(TestA|TestB)$' runs in source order — the regex order
changes nothing (verified, Go 1.26.2). Zero external dependencies: one binary,
stdlib only.
Install
Prebuilt static binaries ship with every release:
linux and darwin (amd64/arm64), windows (amd64), each published with a
SHA256SUMS file.
# latest release (v0.3.0), linux-amd64 — swap the suffix for your platform
curl -fsSLO https://github.com/emrelab/go-flake-hunter/releases/download/v0.3.0/go-flake-hunter-0.3.0-linux-amd64
curl -fsSLO https://github.com/emrelab/go-flake-hunter/releases/download/v0.3.0/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
chmod +x go-flake-hunter-0.3.0-linux-amd64 && mv go-flake-hunter-0.3.0-linux-amd64 go-flake-hunter
# or build it yourself (Go 1.24+)
go install github.com/emrelab/go-flake-hunter/cmd/go-flake-hunter@v0.3.0
The GitHub Action prefers the prebuilt binary of the tag it is pinned to and
falls back to building its own source, so @v0.3.0 needs no toolchain setup
beyond actions/setup-go.
First 5 minutes
go install github.com/emrelab/go-flake-hunter/cmd/go-flake-hunter@latest
# 1) scan every package (default: 1000 runs / 5 min budget)
cd ~/projects/service
go-flake-hunter hunt ./...
# 2) turn the found flake into a permanent test file
go-flake-hunter compile
go test -tags goflakehunter -run '^TestFlakeRegression_' .
# 3) send the capsule to your team (runs no tests, needs no `go`)
go-flake-hunter explain .go-flake-hunter/gfh_*.json --format markdown
Real output (testdata/fixtures/order-basic):
flake found: TestVictim
package : fixture/orderbasic
class : order
polluter : TestPolluter
run : 3/3 failed
Wilson 95% : 43.9%–100.0%
deterministic: yes
evidence : TestPolluter writes, TestVictim reads: seeded
reproduce:
go test . -run '^TestFlakeWitness$' -count=1 -parallel=1
capsule : gfh_e3c71933f905
status : confirmed
The generated regression test (compile):
//go:build goflakehunter
// Code generated by go-flake-hunter; DO NOT EDIT.
func TestFlakeRegression_TestVictim(t *testing.T) {
t.Run("polluter/TestPolluter", TestPolluter)
t.Run("victim/TestVictim", TestVictim)
}
The price of the build tag: go test ./... does not see it. In exchange the
default suite neither slows down nor turns red. The test will be red right
now — the artifact's job is to catch the flake, not to fix it. After the fix it
turns green and guards the regression permanently.
Scanned real repos (beyond our own fixtures)
Three targeted upstream cases completed the full lifecycle:
| Repository / pinned commit |
Minimal polluter → victim |
Result |
gosimple/slug @ 1eab316 |
TestSlugMakeUserSubstituteLang → TestSlugMake |
order; fix merged upstream in #83 (2024), re-validated here |
router-for-me/CLIProxyAPI @ 0074897 |
TestResponsesWebsocketHomeSelectedAuthCallbackPinsAndReusesFirstSelection → TestCodexClientModelsApplyPatchRouting |
order; proposed fix in #7356 (open), applied locally and validated |
Jamf-Concepts/jamf-cli @ 977b704 |
TestResolveAuth_URLNormalization → TestCheckAPIMatchRefusesAPlatformCommandOnAnInstanceProfile |
order; witness + fix proposed in #474 (open), 1873 preceding tests shrunk to one polluter |
The first two (November 10) confirm the lifecycle end to end: confirmed capsule
→ replay 5/5 → generated regression RED → the fix → the same regression GREEN.
Both searches used unmodified upstream test bodies.
The third is the case the tool exists for. A Jamf maintainer had already
reported the order dependence and named the mechanism in
#370; nobody had shrunk
it. repro --from-json on the failing -shuffle log reduced 1873 preceding
tests to a single polluter, compile --verify produced a regression that is red
on main, and the patch is 11 lines using a helper the repository already had.
None of the three was discovered by this project: every flake report was already
public. The contribution is the minimal witness — and, in the jamf-cli case, the
fix — never the diagnosis. The table says which fix is merged and which is still
a proposal.
Commands, exact revisions, capsules and before/after evidence:
real-world validation. These are targeted
historical reproductions, not a measurement of detection rate across arbitrary
repositories. The seven earlier scans below remain recorded as they ran.
What a flake hunter should say about a healthy codebase is nothing — so the
first real-world metric is false positives, not witnesses:
| Repo |
Scan |
Result |
sirupsen/logrus |
root package |
0 flakes; run budget hit → self-reported partial capsule |
rs/zerolog |
root package |
0 flakes; budget-capped, honest partial |
spf13/viper |
root package |
0 flakes; budget-capped, honest partial |
fatih/color |
root package |
0 flakes; budget-capped, honest partial |
mattn/go-colorable |
root package |
clean full scan, 0 flakes |
urfave/cli v3 |
root package |
0 flakes; budget-capped, honest partial |
google/trillian @ pre-#2792 |
./log |
clean full scan, 0 flakes — see note |
The trillian note: we checked out the parent of the commit that fixed
their order-dependent test and hunted ./log there. The suite passed clean on
our machine — their pollution needed a wider run context than the package
replay produces. Reported as-is; no witness claimed.
Across all seven scans: zero findings, so zero false findings. We state it
that way on purpose — seven scans that emitted nothing do not measure a false
positive rate; they only show the hunter stayed quiet on healthy code. Every
witness go-flake-hunter does emit must deterministically replay (that is the
contract the capsule carries). The targeted cases above add witnesses, but
neither they nor these seven scans establish a general false-positive rate.
The two targeted reproductions above complete the real-world witness
milestone (#1); the seven
earlier scans remain evidence only about those specific scans.
Commands
| Command |
What it does |
Runs tests? |
hunt <pkg...> |
scans packages, classifies, writes capsules |
✅ |
repro <pkg> [Test] |
shrinks one known suspect (the budget goes to a single test); --from-json FILE|- takes a failed go test -json log instead of typing suspects |
✅ |
compile [id] [--verify] |
generates a permanent regression test from a capsule; --verify runs the artifact once |
with --verify |
replay [id] |
runs the witness N times, verifies determinism |
✅ |
explain [id] |
renders a capsule as a text/JSON/markdown report |
❌ |
init [dir] |
writes a .github/workflows/go-flake-hunter.yml scaffold |
❌ |
Shared flags: --format text|json|markdown|sarif · --budget N · --timeout 10m ·
--seeds N · --repeat N · --capsule-dir DIR · --fail-on-flake · --race
· -v · --debug (internal error detail) · --short (scan with -short:
the target's own short-skipping tests do not spend the budget). hunt also
takes --changed ·
--base REF · --baseline FILE · --write-baseline · --polluter-pkg PKG
(cross-binary suspect, see below) · --sarif FILE (write SARIF 2.1.0 alongside
any format — one hunt feeds both the job summary and code scanning). compile
also takes --pkg DIR, --print, --dry-run and --verify.
Baselines use commit-independent defect fingerprints (go-flake-hunter/baseline/v2),
so an unrelated commit does not turn a known flake into a new finding. The
fingerprint covers module, victim package/name, class, polluter identities,
environment keys and same-/cross-process shape; run flags and provenance stay
in the capsule. Legacy v1 baselines must be regenerated explicitly with
go-flake-hunter hunt ./... --write-baseline (add --baseline FILE for a custom
path). Scan every previously baselined package: v1 hashes cannot be migrated
without their witnesses. Corrupt baseline files are refused, including during
--write-baseline.
New regression files include the victim name, for example
checkout_TestVictim_goflakehunter_test.go. Compiling a second victim from the
same source file preserves the first. Recompiling one victim updates that
victim's regression; an existing legacy generated file is reused for its own
victim to avoid duplicate declarations. User-owned output files are never
overwritten.
compile --verify keeps three verdicts apart — verified (the red reproduced),
green-not-reproduced and skipped-missing-env — records the one it got in
the capsule's verifyStatus, and exits 3 unless the artifact actually turned
red. A skipped run is never a green verification.
replay makes the same distinction on the other side of the contract: a witness
that no longer fails is exit 3 ("may already be fixed"), a witness whose package
no longer compiles is exit 6. Reading the second as the first is how a
mid-refactor tree deletes a real capsule. The witness function name comes from
the capsule, so a package that already defines TestFlakeWitness replays under
the name the hunt actually used (TestFlakeWitness_2).
--race never narrows the search: the witness is found without race, then
verified once with -race. If it does not reproduce, the capsule's "race"
field stays false and the report adds a "not reproduced under a race run"
note — evidence is never invented.
--format json produces a single document (flakes: [] on a clean run):
{ "schema": "go-flake-hunter/v2", "flakes": [ { "id": "gfh_…", "kind": "order", … } ],
"runs": 42, "maxRuns": 1000 }
go-flake-hunter init --replay-capsule .go-flake-hunter/gfh_ID.json adds a regression
gate for a committed capsule: replayed failure turns CI red, a strictly
non-reproducing witness turns it green, and build/cancellation errors remain
errors. Cross-binary capsules replay both stages; no generated test is needed.
hunt --jobs N scans up to N package directories concurrently (default
min(NumCPU, 4)). All workers share one run/time budget and reports merge in
package order. Use --jobs 1 for repositories whose different packages share
external state. --polluter-pkg scans are serialized so their two-stage
sequences cannot overlap sibling package scans. repro remains serial.
| Capsule status |
Meaning |
CI behavior |
confirmed |
every verification round reproduced |
eligible for --fail-on-flake, baseline and compile |
suspected |
some verification rounds reproduced; reports include k/n and Wilson 95% interval |
advisory SARIF review/note; excluded from gate/baseline; compile refuses |
partial |
scan stopped before a verdict |
exit 4 for budget, 130 for cancellation |
A suspected-only hunt returns 0 with its findings; a suspected-only repro
returns 3 because strict reproduction was not achieved. replay can gather new
observations from a suspected capsule without promoting its stored status.
Intervals summarize the small observed sample; they do not promise a future
failure rate. The JSON Action flakes output counts confirmed findings only.
Exit codes
CI scripts bind to these; no path fails silently with 0.
| Code |
Meaning |
0 |
completed hunt (including advisory suspected findings), or confirmed reproduction (gate off) |
1 |
new confirmed flake found (--fail-on-flake) |
2 |
usage error |
3 |
repro: the requested test did not reproduce within budget; compile --verify: the artifact stayed green or was skipped for a missing env var |
4 |
budget exceeded (partial capsule written) |
5 |
unsupported witness form or compile of a non-confirmed/cross-clause/cross-binary capsule |
6 |
package does not build |
10 |
internal error (detail with --debug) |
130 |
cancelled (SIGINT); half-written witness files deleted |
What it can and cannot find
| Class |
Status |
Witness |
repeat — repeats in the same process |
✅ |
-count=N |
order — another test runs first |
✅ |
polluter→victim chain, shrunk with ddmin |
parallel — only when concurrent |
✅ |
witness with t.Parallel() on every step + -parallel=N |
env — depends on an env var |
✅ |
the key is stored in the capsule, never the value |
subtests (t.Run tree) |
✅ |
the parent test is called; verdicts fold onto subtests |
TestMain |
✅ |
the witness is installed in the same package, TestMain runs too |
| internal/external test clauses in one directory |
✅ order |
one test binary, top-level -run selection and recorded shuffle seed; replay supported, compile refuses because one generated file cannot call both clauses |
| pollution across separate package processes |
⚠️ --polluter-pkg |
only state that survives the process boundary (files, ports): hunt names the two-stage command go test <polluter> && go test <victim>, replay runs both stages. In-memory globals cannot cross two binaries — no witness is claimed. compile refuses these honestly (exit 5) and prints the runnable command |
| time/network/DB dependent |
⚠️ advisory detection |
static clock/network/database/random calls are named on stderr before probing; external services are not controlled or emulated |
Dependency call sites are package-wide syntax hints, not proof of a cause or
an excuse to skip a reproducible test. Aliased imports are recognized, values
are never captured, and dynamic dispatch may be invisible. A synctest bubble
virtualizes its clock but does not make external I/O deterministic. Literal env
differences continue to record keys only and give the concrete replay input.
The application-global fixture matrix in
testdata/fixtures/app-globals measures a static
evidence gap in all three shapes, with no witness recall loss in those three
cases. Deeper production-code analysis remains deferred pending measured need.
GitHub Actions
Listed on the Marketplace as
go-flake-hunter —
the same name as the repository, so the listing and the uses: line below
match.
- uses: emrelab/go-flake-hunter@v0.3.0
with:
paths: ./...
format: markdown
budget: "200"
# fail-on-flake: "true" # if you want the gate
Budget defaults differ by surface, on purpose. Local hunt starts at 1000
runs / 5m — a laptop can afford the coverage of ./...; the Action and the
go-flake-hunter init scaffold start at 200 runs / 10m, because a shared runner's
wall clock is the scarce resource and ten minutes is what a job can spend on one
hunt before it looks hung. Pass budget:/--budget and timeout:/--timeout
when you want the same shape on both.
The action prefers the prebuilt binary of its own tag (verified against
SHA256SUMS) and falls back to building its own source from
GITHUB_ACTION_PATH: pinning a release tag is what says which version is
running, with no waiting on the proxy. Pin a tag, never a branch — @main
means every run of your CI silently executes whatever landed this morning.
Go 1.24+ is required
(actions/setup-go is on you).
Turn an already-red go test log into a PR comment — repro --from-json shrinks
the log's own failures to a witness, no re-scan: docs/CI-REPRO.md.
go-flake-hunter init writes this file into your repo (it never overwrites an
existing one). The default is soft mode: a newly adopted flake hunter that
turns CI red on day one gets removed on day two.
Design
- The witness is evidence, not a fix: the generated test does not fix the
flake, it reproduces it. It turns green after the fix.
- Noise-free: the witness file gets a
zz_ prefix (Go skips _/.
prefixed files) and is deleted after the run; compile output is build-tagged.
- Budgeted: every run counts, cancellation works; an interrupted run is
never read as "passed".
- Leak-proof: the env key is stored, the value is not.
Full design: docs/ARCHITECTURE.md.
Development
make test # go test -race ./...
make build # ./go-flake-hunter
make fmt # gofmt
Every module under testdata/fixtures demonstrates a single flake class;
adding a new class is the easiest entry point. See CONTRIBUTING.md.
CI dogfoods the tool on its own repo: the self-hunt job runs go-flake-hunter hunt against go-flake-hunter's packages, so a leaked witness file or a global
state leak turns our CI red before it could embarrass a user.
License
Apache-2.0 — LICENSE.