README
¶
testrig
A Go test harness for running your tests with universal setup and teardown, with a focus on hunting and fixing flaky tests.
Quickstart: Find Flaky Tests
go install github.com/smartcontractkit/testrig/cmd/testrig@latest # Install
testrig diagnose --iterations 5 -- ./... # Run your test suite 5 times and see detailed stats
How Many iterations Do I Need?
If you want to be sure that you have fixed a flaky test, or are chasing down a rare flake scenario, you'll need to re-run the test a bunch of times. The math gets a little complicated on exactly how many, so here's a simplified table to show you how confident you should be in your results.
| Iterations | Chance you missed a flake |
|---|---|
| 5 | 50% |
| 30 | 10% |
| 60 | 5% |
| 150 | 2% |
| 300 | 1% |
| 500+ | < 1% |
Run
testrig enables a few QoL features for running large Go test suites efficiently. You can define test lifecycle hooks, and run from the CLI, or from a lightweight Go setup.
Lifecycle Hooks
Run setup and teardown scripts during your test lifecycle.
| Option | When it runs | CLI equivalent |
|---|---|---|
testrig.GlobalSetup |
Run once before any tests | --global-setup |
testrig.GlobalTeardown |
Run once after all tests finish | --global-teardown |
testrig.IterationSetup |
Run before each diagnose iteration | --iteration-setup |
testrig.IterationTeardown |
Run after each diagnose iteration | --iteration-teardown |
CLI
# Spin up dependencies before any test, tear down after
testrig diagnose --iterations 10 \
--global-setup "docker compose up -d" \
--global-teardown "docker compose down" \
-- ./...
# Reset DB state between each diagnose iteration
testrig diagnose --iterations 10 \
--iteration-setup "psql -c 'TRUNCATE events'" \
--iteration-teardown "rm -rf ./tmp/artifacts" \
-- ./...
Native Go
You can import testrig as a Go package and define your hooks and defaults entirely in Go! See the pkg.go.dev docs for an example setup.
Documentation
¶
Overview ¶
Package testrig is a Go test harness for hunting flaky tests. It exposes lifecycle hooks (GlobalSetup/Teardown, IterationSetup/Teardown) that callers wire into TestMain so the same hooks work whether the test suite is invoked directly via go test or under the testrig diagnose CLI.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Run ¶
func Run(opts ...Option)
Run is the entry point for running the testrig test suite from a test binary. Calling Run executes the testrig CLI engine and will call os.Exit upon completion.
Example ¶
package main
import (
"context"
"fmt"
"github.com/smartcontractkit/testrig"
)
func main() {
testrig.Run(
// GlobalSetup runs once before any tests start.
testrig.GlobalSetup(func(_ context.Context) error {
fmt.Println("Starting mock background service...")
return nil
}),
// IterationSetup runs before each diagnose iteration.
testrig.IterationSetup(func(_ context.Context) error {
fmt.Println("Resetting database state for next iteration...")
return nil
}),
// IterationTeardown runs after each diagnose iteration.
testrig.IterationTeardown(func(_ context.Context) error {
fmt.Println("Cleaning up iteration artifacts...")
return nil
}),
// GlobalTeardown runs once after all tests finish.
testrig.GlobalTeardown(func(_ context.Context) error {
fmt.Println("Stopping mock background service...")
return nil
}),
)
}
Output:
Types ¶
type Hook ¶
Hook is a lifecycle callback. The context carries cancellation from the test runner — hooks should respect it for long-running operations.
func NewShellHook ¶
NewShellHook returns a Hook that runs cmdStr via the system shell (sh -c). The hook respects context cancellation. On non-zero exit, the combined stdout+stderr is included in the returned error so failing setup commands are diagnosable.
type Option ¶
Option configures the testrig CLI runner.
func GlobalSetup ¶
GlobalSetup registers a hook to run once before any tests.
func GlobalTeardown ¶
GlobalTeardown registers a hook to run once after all tests finish.
func IterationSetup ¶
IterationSetup registers a hook to run before each diagnose iteration.
func IterationTeardown ¶
IterationTeardown registers a hook to run after each diagnose iteration.
type RunOptions ¶
type RunOptions = hooks.RunOptions
RunOptions contains the evaluated configuration for the testrig CLI. It is exported for internal use by the CLI engine.
func BuildOptions ¶
func BuildOptions(opts ...Option) RunOptions
BuildOptions evaluates the functional options and returns the internal struct. It is exported for internal use by the CLI engine.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
testrig
command
Package main is the testrig CLI entry point.
|
Package main is the testrig CLI entry point. |
|
internal
|
|
|
assets
Package assets contains embedded files and templates.
|
Package assets contains embedded files and templates. |
|
cmd
Package cmd implements the testrig CLI commands.
|
Package cmd implements the testrig CLI commands. |
|
config
Package config holds the harness's flag-bound application config.
|
Package config holds the harness's flag-bound application config. |
|
hooks
Package hooks defines the testrig lifecycle hook interfaces and configuration.
|
Package hooks defines the testrig lifecycle hook interfaces and configuration. |
|
output
Package output centralizes harness-owned terminal writes: human-rich vs sparse (--ai-output), and whether stderr supports carriage-return progress (TTY).
|
Package output centralizes harness-owned terminal writes: human-rich vs sparse (--ai-output), and whether stderr supports carriage-return progress (TTY). |
|
runner
Package runner implements go test wrappers, iteration orchestration, and result analysis.
|
Package runner implements go test wrappers, iteration orchestration, and result analysis. |
|
termstyle
Package termstyle holds lipgloss styles shared by diagnose progress, DB setup, and summary output so the CLI reads as one palette.
|
Package termstyle holds lipgloss styles shared by diagnose progress, DB setup, and summary output so the CLI reads as one palette. |
|
tools
|
|
|
gendocs
command
Command gendocs generates the lifecycle hooks reference table from hooks.go godoc and splices it into HOOKS.md and README files.
|
Command gendocs generates the lifecycle hooks reference table from hooks.go godoc and splices it into HOOKS.md and README files. |