testrig

README

testrig

A Go test harness for running your tests with universal setup and teardown, with a focus on hunting and fixing flaky tests.

Quickstart: Find Flaky Tests

go install github.com/smartcontractkit/testrig/cmd/testrig@latest # Install
testrig diagnose --iterations 5 -- ./... # Run your test suite 5 times and see detailed stats

How Many iterations Do I Need?

If you want to be sure that you have fixed a flaky test, or are chasing down a rare flake scenario, you'll need to re-run the test a bunch of times. The math gets a little complicated on exactly how many, so here's a simplified table to show you how confident you should be in your results.

Iterations	Chance you missed a flake
5	50%
30	10%
60	5%
150	2%
300	1%
500+	< 1%

Run

testrig enables a few QoL features for running large Go test suites efficiently. You can define test lifecycle hooks, and run from the CLI, or from a lightweight Go setup.

Lifecycle Hooks

Run setup and teardown scripts during your test lifecycle.

Option	When it runs	CLI equivalent
testrig.GlobalSetup	Run once before any tests	--global-setup
testrig.GlobalTeardown	Run once after all tests finish	--global-teardown
testrig.IterationSetup	Run before each diagnose iteration	--iteration-setup
testrig.IterationTeardown	Run after each diagnose iteration	--iteration-teardown

CLI

# Vanilla go test with global hooks
testrig --global-setup "docker compose up -d" -v -count=1 ./...

# Spin up dependencies before any test, tear down after
testrig diagnose --iterations 10 \
  --global-setup "docker compose up -d" \
  --global-teardown "docker compose down" \
  -- ./...

# Reset DB state between each diagnose iteration
testrig diagnose --iterations 10 \
  --iteration-setup "psql -c 'TRUNCATE events'" \
  --iteration-teardown "rm -rf ./tmp/artifacts" \
  -- ./...

Native Go

You can import testrig as a Go package and define your hooks and defaults entirely in Go! See the pkg.go.dev docs for an example setup.

Documentation

Overview

Package testrig is a Go test harness for hunting flaky tests. It exposes lifecycle hooks (GlobalSetup/Teardown, IterationSetup/Teardown) that callers wire into TestMain so the same hooks work whether the test suite is invoked directly via go test or under the testrig diagnose CLI.

Index

• func Run(opts ...Option)
• type Hook
  • func NewShellHook(cmdStr string) Hook
• type Option
  • func GlobalSetup(h Hook) Option
  • func GlobalTeardown(h Hook) Option
  • func IterationSetup(h Hook) Option
  • func IterationTeardown(h Hook) Option
  • func WithCommand(cmd *cobra.Command) Option
  • func WithResources(p ResourceProvider) Option
  • func WithRootCommand(name string) Option
  • func WithRootFlags(register func(*pflag.FlagSet)) Option
• type Resource
• type ResourceProvider
• type RunOptions
  • func BuildOptions(opts ...Option) RunOptions

Examples

• Run

Functions

func Run

func Run(opts ...Option)

Run is the entry point for running the testrig test suite from a test binary. Calling Run executes the testrig CLI engine and will call os.Exit upon completion.

Example

package main

import (
	"context"
	"fmt"

	"github.com/smartcontractkit/testrig"
)

func main() {
	testrig.Run(
		// GlobalSetup runs once before any tests start.
		testrig.GlobalSetup(func(_ context.Context) error {
			fmt.Println("Starting mock background service...")
			return nil
		}),

		// IterationSetup runs before each diagnose iteration.
		testrig.IterationSetup(func(_ context.Context) error {
			fmt.Println("Resetting database state for next iteration...")
			return nil
		}),

		// IterationTeardown runs after each diagnose iteration.
		testrig.IterationTeardown(func(_ context.Context) error {
			fmt.Println("Cleaning up iteration artifacts...")
			return nil
		}),

		// GlobalTeardown runs once after all tests finish.
		testrig.GlobalTeardown(func(_ context.Context) error {
			fmt.Println("Stopping mock background service...")
			return nil
		}),
	)
}

Types

type Hook

type Hook = hooks.Hook

Hook is a lifecycle callback. The context carries cancellation from the test runner — hooks should respect it for long-running operations.

func NewShellHook

func NewShellHook(cmdStr string) Hook

NewShellHook returns a Hook that runs cmdStr via the system shell (sh -c). The hook respects context cancellation. On non-zero exit, the combined stdout+stderr is included in the returned error so failing setup commands are diagnosable.

type Option

type Option = hooks.Option

Option configures the testrig CLI runner.

func GlobalSetup

func GlobalSetup(h Hook) Option

GlobalSetup registers a hook to run once before any tests.

func GlobalTeardown

func GlobalTeardown(h Hook) Option

GlobalTeardown registers a hook to run once after all tests finish.

func IterationSetup

func IterationSetup(h Hook) Option

IterationSetup registers a hook to run before each diagnose iteration.

func IterationTeardown

func IterationTeardown(h Hook) Option

IterationTeardown registers a hook to run after each diagnose iteration.

func WithCommand

func WithCommand(cmd *cobra.Command) Option

WithCommand registers an additional subcommand on the testrig root command, for project-specific utilities (e.g. persistent database management).

func WithResources

func WithResources(p ResourceProvider) Option

WithResources registers a provider that supplies isolated infrastructure (e.g. databases) for the run. See ResourceProvider for count semantics and partial-failure behavior. Each resource's Env is appended to the child go test process; Reset and DumpDiagnostics apply during diagnose; Cleanup runs once after the command finishes.

func WithRootCommand

func WithRootCommand(name string) Option

WithRootCommand sets the CLI name used in help text, examples, and cobra.CommandPath(). Defaults to "testrig" when unset.

func WithRootFlags

func WithRootFlags(register func(*pflag.FlagSet)) Option

WithRootFlags registers persistent flags on the root command, available to every subcommand. Use it to add consumer flags (e.g. --database-url) that a resource provider or custom command reads.

type Resource

type Resource = hooks.Resource

Resource is one prepared, isolated piece of infrastructure (e.g. a database) supplied by a ResourceProvider. See hooks.Resource for field semantics.

type ResourceProvider

type ResourceProvider = hooks.ResourceProvider

ResourceProvider supplies isolated resources for a run. See WithResources.

type RunOptions

type RunOptions = hooks.RunOptions

RunOptions contains the evaluated configuration for the testrig CLI. It is exported for internal use by the CLI engine.

func BuildOptions

func BuildOptions(opts ...Option) RunOptions

BuildOptions evaluates the functional options and returns the internal struct. It is exported for internal use by the CLI engine.