go-flake-hunter

module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: Apache-2.0

README

go-flake-hunter

ci nightly Marketplace Go Reference Go Report Card license

Reproduce the flake in 30 seconds instead of 30 minutes — and leave a permanent regression test in the repo.

257 tests · CI on linux/macos/windows + a nightly sweep · zero dependencies · three real upstream flake cases reduced to witnesses (evidence)

Finds flaky Go-test failures caused by order, parallelism, repetition and environment, shrinks them to the smallest witness (polluter → victim) and generates a regression test you can commit.

It does not detect flaky tests — that layer is crowded (BuildPulse, Trunk, gotestsum --rerun-fails). This tool does reproduction and permanence: it turns a failing behavior into a test file that never gets lost again.

go test -json  →  hypothesis order  →  ddmin  →  capsule  →  regression test
                  repeat             smallest
                  order              polluter→victim
                  parallel
                  env
Why not just -count=100?
Approach What it gives you What is left for you
go test -count=100 / -shuffle=on a red log, eventually explain the failure, keep the flake
gotestsum --rerun-fails a rerun of failed tests the flake is still in the repo
stress, -parallel loops a higher reproduction probability still no minimal witness
go-flake-hunter polluter→victim chain, shrunk with ddmin, written as a build-tagged regression test the fix

Detection is not the product; permanence is. A tracker tells you a test is flaky. go-flake-hunter hands you a test file that stays red until the cause is gone.

go-flake-hunter in action

Changing the order requires a temporary witness file, because in Go go test -run '^(TestA|TestB)$' runs in source order — the regex order changes nothing (verified, Go 1.26.2). Zero external dependencies: one binary, stdlib only.

Install

Prebuilt static binaries ship with every release: linux and darwin (amd64/arm64), windows (amd64), each published with a SHA256SUMS file.

# latest release (v0.3.0), linux-amd64 — swap the suffix for your platform
curl -fsSLO https://github.com/emrelab/go-flake-hunter/releases/download/v0.3.0/go-flake-hunter-0.3.0-linux-amd64
curl -fsSLO https://github.com/emrelab/go-flake-hunter/releases/download/v0.3.0/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
chmod +x go-flake-hunter-0.3.0-linux-amd64 && mv go-flake-hunter-0.3.0-linux-amd64 go-flake-hunter

# or build it yourself (Go 1.24+)
go install github.com/emrelab/go-flake-hunter/cmd/go-flake-hunter@v0.3.0

The GitHub Action prefers the prebuilt binary of the tag it is pinned to and falls back to building its own source, so @v0.3.0 needs no toolchain setup beyond actions/setup-go.

First 5 minutes

go install github.com/emrelab/go-flake-hunter/cmd/go-flake-hunter@latest

# 1) scan every package (default: 1000 runs / 5 min budget)
cd ~/projects/service
go-flake-hunter hunt ./...

# 2) turn the found flake into a permanent test file
go-flake-hunter compile
go test -tags goflakehunter -run '^TestFlakeRegression_' .

# 3) send the capsule to your team (runs no tests, needs no `go`)
go-flake-hunter explain .go-flake-hunter/gfh_*.json --format markdown

Real output (testdata/fixtures/order-basic):

flake found: TestVictim
  package      : fixture/orderbasic
  class        : order
  polluter     : TestPolluter
  run          : 3/3 failed
  Wilson 95%   : 43.9%–100.0%
  deterministic: yes
  evidence     : TestPolluter writes, TestVictim reads: seeded

  reproduce:
    go test . -run '^TestFlakeWitness$' -count=1 -parallel=1

  capsule      : gfh_e3c71933f905
  status       : confirmed

The generated regression test (compile):

//go:build goflakehunter

// Code generated by go-flake-hunter; DO NOT EDIT.
func TestFlakeRegression_TestVictim(t *testing.T) {
	t.Run("polluter/TestPolluter", TestPolluter)
	t.Run("victim/TestVictim", TestVictim)
}

The price of the build tag: go test ./... does not see it. In exchange the default suite neither slows down nor turns red. The test will be red right now — the artifact's job is to catch the flake, not to fix it. After the fix it turns green and guards the regression permanently.

Scanned real repos (beyond our own fixtures)

Three targeted upstream cases completed the full lifecycle:

Repository / pinned commit Minimal polluter → victim Result
gosimple/slug @ 1eab316 TestSlugMakeUserSubstituteLang → TestSlugMake order; fix merged upstream in #83 (2024), re-validated here
router-for-me/CLIProxyAPI @ 0074897 TestResponsesWebsocketHomeSelectedAuthCallbackPinsAndReusesFirstSelection → TestCodexClientModelsApplyPatchRouting order; proposed fix in #7356 (open), applied locally and validated
Jamf-Concepts/jamf-cli @ 977b704 TestResolveAuth_URLNormalization → TestCheckAPIMatchRefusesAPlatformCommandOnAnInstanceProfile order; witness + fix proposed in #474 (open), 1873 preceding tests shrunk to one polluter

The first two (November 10) confirm the lifecycle end to end: confirmed capsule → replay 5/5 → generated regression RED → the fix → the same regression GREEN. Both searches used unmodified upstream test bodies.

The third is the case the tool exists for. A Jamf maintainer had already reported the order dependence and named the mechanism in #370; nobody had shrunk it. repro --from-json on the failing -shuffle log reduced 1873 preceding tests to a single polluter, compile --verify produced a regression that is red on main, and the patch is 11 lines using a helper the repository already had.

None of the three was discovered by this project: every flake report was already public. The contribution is the minimal witness — and, in the jamf-cli case, the fix — never the diagnosis. The table says which fix is merged and which is still a proposal.

Commands, exact revisions, capsules and before/after evidence: real-world validation. These are targeted historical reproductions, not a measurement of detection rate across arbitrary repositories. The seven earlier scans below remain recorded as they ran.

What a flake hunter should say about a healthy codebase is nothing — so the first real-world metric is false positives, not witnesses:

Repo Scan Result
sirupsen/logrus root package 0 flakes; run budget hit → self-reported partial capsule
rs/zerolog root package 0 flakes; budget-capped, honest partial
spf13/viper root package 0 flakes; budget-capped, honest partial
fatih/color root package 0 flakes; budget-capped, honest partial
mattn/go-colorable root package clean full scan, 0 flakes
urfave/cli v3 root package 0 flakes; budget-capped, honest partial
google/trillian @ pre-#2792 ./log clean full scan, 0 flakes — see note

The trillian note: we checked out the parent of the commit that fixed their order-dependent test and hunted ./log there. The suite passed clean on our machine — their pollution needed a wider run context than the package replay produces. Reported as-is; no witness claimed.

Across all seven scans: zero findings, so zero false findings. We state it that way on purpose — seven scans that emitted nothing do not measure a false positive rate; they only show the hunter stayed quiet on healthy code. Every witness go-flake-hunter does emit must deterministically replay (that is the contract the capsule carries). The targeted cases above add witnesses, but neither they nor these seven scans establish a general false-positive rate. The two targeted reproductions above complete the real-world witness milestone (#1); the seven earlier scans remain evidence only about those specific scans.

Commands

Command What it does Runs tests?
hunt <pkg...> scans packages, classifies, writes capsules ✅
repro <pkg> [Test] shrinks one known suspect (the budget goes to a single test); --from-json FILE|- takes a failed go test -json log instead of typing suspects ✅
compile [id] [--verify] generates a permanent regression test from a capsule; --verify runs the artifact once with --verify
replay [id] runs the witness N times, verifies determinism ✅
explain [id] renders a capsule as a text/JSON/markdown report ❌
init [dir] writes a .github/workflows/go-flake-hunter.yml scaffold ❌

Shared flags: --format text|json|markdown|sarif · --budget N · --timeout 10m · --seeds N · --repeat N · --capsule-dir DIR · --fail-on-flake · --race · -v · --debug (internal error detail) · --short (scan with -short: the target's own short-skipping tests do not spend the budget). hunt also takes --changed · --base REF · --baseline FILE · --write-baseline · --polluter-pkg PKG (cross-binary suspect, see below) · --sarif FILE (write SARIF 2.1.0 alongside any format — one hunt feeds both the job summary and code scanning). compile also takes --pkg DIR, --print, --dry-run and --verify.

Baselines use commit-independent defect fingerprints (go-flake-hunter/baseline/v2), so an unrelated commit does not turn a known flake into a new finding. The fingerprint covers module, victim package/name, class, polluter identities, environment keys and same-/cross-process shape; run flags and provenance stay in the capsule. Legacy v1 baselines must be regenerated explicitly with go-flake-hunter hunt ./... --write-baseline (add --baseline FILE for a custom path). Scan every previously baselined package: v1 hashes cannot be migrated without their witnesses. Corrupt baseline files are refused, including during --write-baseline.

New regression files include the victim name, for example checkout_TestVictim_goflakehunter_test.go. Compiling a second victim from the same source file preserves the first. Recompiling one victim updates that victim's regression; an existing legacy generated file is reused for its own victim to avoid duplicate declarations. User-owned output files are never overwritten.

compile --verify keeps three verdicts apart — verified (the red reproduced), green-not-reproduced and skipped-missing-env — records the one it got in the capsule's verifyStatus, and exits 3 unless the artifact actually turned red. A skipped run is never a green verification.

replay makes the same distinction on the other side of the contract: a witness that no longer fails is exit 3 ("may already be fixed"), a witness whose package no longer compiles is exit 6. Reading the second as the first is how a mid-refactor tree deletes a real capsule. The witness function name comes from the capsule, so a package that already defines TestFlakeWitness replays under the name the hunt actually used (TestFlakeWitness_2).

--race never narrows the search: the witness is found without race, then verified once with -race. If it does not reproduce, the capsule's "race" field stays false and the report adds a "not reproduced under a race run" note — evidence is never invented.

--format json produces a single document (flakes: [] on a clean run):

{ "schema": "go-flake-hunter/v2", "flakes": [ { "id": "gfh_…", "kind": "order", … } ],
  "runs": 42, "maxRuns": 1000 }

go-flake-hunter init --replay-capsule .go-flake-hunter/gfh_ID.json adds a regression gate for a committed capsule: replayed failure turns CI red, a strictly non-reproducing witness turns it green, and build/cancellation errors remain errors. Cross-binary capsules replay both stages; no generated test is needed.

hunt --jobs N scans up to N package directories concurrently (default min(NumCPU, 4)). All workers share one run/time budget and reports merge in package order. Use --jobs 1 for repositories whose different packages share external state. --polluter-pkg scans are serialized so their two-stage sequences cannot overlap sibling package scans. repro remains serial.

Capsule status Meaning CI behavior
confirmed every verification round reproduced eligible for --fail-on-flake, baseline and compile
suspected some verification rounds reproduced; reports include k/n and Wilson 95% interval advisory SARIF review/note; excluded from gate/baseline; compile refuses
partial scan stopped before a verdict exit 4 for budget, 130 for cancellation

A suspected-only hunt returns 0 with its findings; a suspected-only repro returns 3 because strict reproduction was not achieved. replay can gather new observations from a suspected capsule without promoting its stored status. Intervals summarize the small observed sample; they do not promise a future failure rate. The JSON Action flakes output counts confirmed findings only.

Exit codes

CI scripts bind to these; no path fails silently with 0.

Code Meaning
0 completed hunt (including advisory suspected findings), or confirmed reproduction (gate off)
1 new confirmed flake found (--fail-on-flake)
2 usage error
3 repro: the requested test did not reproduce within budget; compile --verify: the artifact stayed green or was skipped for a missing env var
4 budget exceeded (partial capsule written)
5 unsupported witness form or compile of a non-confirmed/cross-clause/cross-binary capsule
6 package does not build
10 internal error (detail with --debug)
130 cancelled (SIGINT); half-written witness files deleted

What it can and cannot find

Class Status Witness
repeat — repeats in the same process ✅ -count=N
order — another test runs first ✅ polluter→victim chain, shrunk with ddmin
parallel — only when concurrent ✅ witness with t.Parallel() on every step + -parallel=N
env — depends on an env var ✅ the key is stored in the capsule, never the value
subtests (t.Run tree) ✅ the parent test is called; verdicts fold onto subtests
TestMain ✅ the witness is installed in the same package, TestMain runs too
internal/external test clauses in one directory ✅ order one test binary, top-level -run selection and recorded shuffle seed; replay supported, compile refuses because one generated file cannot call both clauses
pollution across separate package processes ⚠️ --polluter-pkg only state that survives the process boundary (files, ports): hunt names the two-stage command go test <polluter> && go test <victim>, replay runs both stages. In-memory globals cannot cross two binaries — no witness is claimed. compile refuses these honestly (exit 5) and prints the runnable command
time/network/DB dependent ⚠️ advisory detection static clock/network/database/random calls are named on stderr before probing; external services are not controlled or emulated

Dependency call sites are package-wide syntax hints, not proof of a cause or an excuse to skip a reproducible test. Aliased imports are recognized, values are never captured, and dynamic dispatch may be invisible. A synctest bubble virtualizes its clock but does not make external I/O deterministic. Literal env differences continue to record keys only and give the concrete replay input.

The application-global fixture matrix in testdata/fixtures/app-globals measures a static evidence gap in all three shapes, with no witness recall loss in those three cases. Deeper production-code analysis remains deferred pending measured need.

GitHub Actions

Listed on the Marketplace as go-flake-hunter — the same name as the repository, so the listing and the uses: line below match.

- uses: emrelab/go-flake-hunter@v0.3.0
  with:
    paths: ./...
    format: markdown
    budget: "200"
    # fail-on-flake: "true"   # if you want the gate

Budget defaults differ by surface, on purpose. Local hunt starts at 1000 runs / 5m — a laptop can afford the coverage of ./...; the Action and the go-flake-hunter init scaffold start at 200 runs / 10m, because a shared runner's wall clock is the scarce resource and ten minutes is what a job can spend on one hunt before it looks hung. Pass budget:/--budget and timeout:/--timeout when you want the same shape on both.

The action prefers the prebuilt binary of its own tag (verified against SHA256SUMS) and falls back to building its own source from GITHUB_ACTION_PATH: pinning a release tag is what says which version is running, with no waiting on the proxy. Pin a tag, never a branch — @main means every run of your CI silently executes whatever landed this morning. Go 1.24+ is required (actions/setup-go is on you).

Turn an already-red go test log into a PR comment — repro --from-json shrinks the log's own failures to a witness, no re-scan: docs/CI-REPRO.md.

go-flake-hunter init writes this file into your repo (it never overwrites an existing one). The default is soft mode: a newly adopted flake hunter that turns CI red on day one gets removed on day two.

Design

  • The witness is evidence, not a fix: the generated test does not fix the flake, it reproduces it. It turns green after the fix.
  • Noise-free: the witness file gets a zz_ prefix (Go skips _/. prefixed files) and is deleted after the run; compile output is build-tagged.
  • Budgeted: every run counts, cancellation works; an interrupted run is never read as "passed".
  • Leak-proof: the env key is stored, the value is not.

Full design: docs/ARCHITECTURE.md.

Development

make test    # go test -race ./...
make build   # ./go-flake-hunter
make fmt     # gofmt

Every module under testdata/fixtures demonstrates a single flake class; adding a new class is the easiest entry point. See CONTRIBUTING.md.

CI dogfoods the tool on its own repo: the self-hunt job runs go-flake-hunter hunt against go-flake-hunter's packages, so a leaked witness file or a global state leak turns our CI red before it could embarrass a user.

License

Apache-2.0 — LICENSE.

Directories

Path Synopsis
cmd
go-flake-hunter command
Command go-flake-hunter reproduces flakes in Go tests and generates a regression test that can be committed to the repository.
Command go-flake-hunter reproduces flakes in Go tests and generates a regression test that can be committed to the repository.
internal
capsule
Package capsule writes and reads the persistent record of a witness.
Package capsule writes and reads the persistent record of a witness.
cli
Package cli manages the command-line surface and the exit codes.
Package cli manages the command-line surface and the exit codes.
cli/compile
Package clicompile holds the compile tests: the three output surfaces (--print, --dry-run, write), the package-target rule, and --verify, which proves the generated regression test actually goes red.
Package clicompile holds the compile tests: the three output surfaces (--print, --dry-run, write), the package-target rule, and --verify, which proves the generated regression test actually goes red.
cli/hunt
Package clihunt holds the hunt engine's integration tests: end-to-end runs against the fixtures, the JSON document contract, and the flags a user types.
Package clihunt holds the hunt engine's integration tests: end-to-end runs against the fixtures, the JSON document contract, and the flags a user types.
cli/huntci
Package clihuntci holds the tests for hunt's CI-facing subset: the baseline file (a known flake must not break the gate) and --changed (only the packages a diff touched).
Package clihuntci holds the tests for hunt's CI-facing subset: the baseline file (a known flake must not break the gate) and --changed (only the packages a diff touched).
cli/huntevidence
Package clihuntevidence holds the tests for what hunt reports beyond the bare finding: the race label, the SARIF document, the cross-binary (two-process) path, and the parallel-race fixture contract that every concurrency claim is calibrated against.
Package clihuntevidence holds the tests for what hunt reports beyond the bare finding: the race label, the SARIF document, the cross-binary (two-process) path, and the parallel-race fixture contract that every concurrency claim is calibrated against.
cli/replay
Package clireplay holds the replay and repro tests: both take a recorded artifact — a capsule, or a failed `go test -json` log — and re-execute it, which is why they share a package.
Package clireplay holds the replay and repro tests: both take a recorded artifact — a capsule, or a failed `go test -json` log — and re-execute it, which is why they share a package.
cli/surface
Package clisurface holds the CLI's own surface tests: flags, help text, usage errors, init and explain.
Package clisurface holds the CLI's own surface tests: flags, help text, usage errors, init and explain.
clitest
Package clitest holds the shared harness for the CLI's integration tests.
Package clitest holds the shared harness for the CLI's integration tests.
compile
Package compile turns a capsule into a permanent regression test.
Package compile turns a capsule into a permanent regression test.
evidence
Package evidence collects observations for a witness (env, files, ports, globals, race).
Package evidence collects observations for a witness (env, files, ports, globals, race).
facts
Package facts extracts the static facts of a test package.
Package facts extracts the static facts of a test package.
gotest
Package gotest runs `go test` and parses its JSON event stream.
Package gotest runs `go test` and parses its JSON event stream.
harness
Package harness renders, installs and removes the temporary witness test file.
Package harness renders, installs and removes the temporary witness test file.
policy
Package policy: budget, cancellation and secret redaction.
Package policy: budget, cancellation and secret redaction.
probe
Package probe tries the four flake hypotheses in cost order.
Package probe tries the four flake hypotheses in cost order.
report
Package report formats a witness for humans and for CI.
Package report formats a witness for humans and for CI.
search
Package search shrinks a candidate set to the smallest witness.
Package search shrinks a candidate set to the smallest witness.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL