metatesting

Documentation

Overview

Package metatesting is a package for writing normal go tests that invoke gosim tests. Such metatests can run gosim tests with different seeds, assert that some scenarios happen often or never, etc. With metatesting, gosim tests can be integrated in a normal go test run, so that 'go test ./...' for a module can cover both simulated and non-simulated tests.

Using build constraints to combine gosim tests and metatests

The metatesting API does not work inside gosim. To combine gosim tests and metatests in a single package, use the '//go:build sim' and '//go:build !sim' contraints at the top of test files.

Working with the go test cache

Metatesting has some unfortunate interactions with go test cache. To run metatests using 'go test', the run needs either be non-cached or the metatests need to be precompiled.

To run metatests with a non-cached 'go test' invocation, run 'go test -count=1 ./package/with/metatests'. Then metatesting will build the gosim tests when needed. This is convenient, but this metatest will not be cached.

To cache tests, the gosim tests need to be built before running the metatest with 'gosim build-tests'. If metatesting is used incorrectly it tries to give a helpful error message. A simple way to run tests is to use a script that invoke 'gosim build-tests' before running 'go test':

# build gosim test binaries
go run github.com/jellevandenhooff/gosim/cmd/gosim build-tests pkgA pkgA/pkgB
# then run tests
go test ./...

Tests need to be precompiled when using the test cache because the test cache should be invalidated when the gosim test changes. Normal go tests get invalidated when their files change because the go tool knows which files are compiled into the test. However, for a metatest that invokes 'gosim test' the go tool does not know what files 'gosim test' accesses, and so does not invalidate the cache if those files change. To sidestep those issues, run 'gosim build-tests' before running 'go test'.

TODO: Support fuzzing.

TODO: Support parallelism.

Index

• func CheckDeterministic(t *testing.T, mt *MetaT)
• func CheckSeeds(t *testing.T, mt *MetaT, numSeeds int)
• func MustFindLogValue(logs []map[string]any, msg string, key string) any
• func ParseLog(logs []byte) []map[string]any
• func SimplifyParsedLog(logs []map[string]any) []string
• type MetaT
  • func ForCurrentPackage(t *testing.T) *MetaT
  • func ForOtherPackage(t *testing.T, pkg string) *MetaT
  • func (r *MetaT) ListTests() ([]string, error)
  • func (r *MetaT) Run(t *testing.T, config *RunConfig) (*RunResult, error)
  • func (r *MetaT) RunAllTests(t *testing.T)
• type RunConfig
• type RunResult

Functions

func CheckDeterministic

func CheckDeterministic(t *testing.T, mt *MetaT)

func CheckSeeds

func CheckSeeds(t *testing.T, mt *MetaT, numSeeds int)

func MustFindLogValue

func MustFindLogValue(logs []map[string]any, msg string, key string) any

func ParseLog

func ParseLog(logs []byte) []map[string]any

func SimplifyParsedLog

func SimplifyParsedLog(logs []map[string]any) []string

Types

type MetaT

type MetaT struct {
	// contains filtered or unexported fields
}

A MetaT runs gosim tests for a given package. It can run tests with various flags and returns their results.

Behind the scenes a MetaT executes and controls a gosim test binary.

func ForCurrentPackage

func ForCurrentPackage(t *testing.T) *MetaT

ForCurrentPackage returns a *MetaT for the package currently being tested by `go test`.

ForCurrentPackage relies on the working directory to identify the current package, so if a test changes directories ForCurrentPackage will not work correctly.

func ForOtherPackage

func ForOtherPackage(t *testing.T, pkg string) *MetaT

ForOtherPackage retursn a *MetaT for the specified package.

func (*MetaT) ListTests

func (r *MetaT) ListTests() ([]string, error)

TODO: Support match expressions?

func (*MetaT) Run

func (r *MetaT) Run(t *testing.T, config *RunConfig) (*RunResult, error)

func (*MetaT) RunAllTests

func (r *MetaT) RunAllTests(t *testing.T)

type RunConfig

type RunConfig struct {
	Test     string
	Seed     int64
	ExtraEnv []string
	Simtrace string
}

A RunConfig configures a test invocation.

type RunResult

type RunResult struct {
	Seed      int64
	Checksum  []byte
	Failed    bool
	LogOutput []byte
	Err       string // TODO: reconsider this type?
}

A RunResult contains the result fo a test run.