testy

README

Testy

A Go test running framework.

Please use the reference badge above to find the full documentation of this package.

We couldn't find a framework that addressed our API acceptance testing desires, so we're making one ourselves. We had some specific design goals in mind when we started this project:

• Maintain the ability to run tests via go test during development of tests, including its ability to run specific tests only.
• Provide the ability to run tests as a service, so that they may be run on a schedule and as part of CI/CD.
• The same tests should be able to be run both ways.
• Tests should be written in a familiar manner.
• Historical test results should be stored somewhere.

As this is intended for external API acceptance tests, it is expected that the test code itself does not reside in the same repository as the code under test. We have the tests for all of our external APIs in a single test repository, even though those tests are testing several different APIs. This makes it easier to run tests over all external APIs at the same time, to ensure a change to an internal service that is used by multiple APIs does not cause any such API to fail.

Examples

Please see the Example directory.

Documentation

Overview

Package testy is a Go test running framework.

Index

• Variables
• func AddEchoRoutes(router *echo.Group)
• func AfterPackage(f Tester) any
• func AfterTest(f Tester) any
• func BeforePackage(f Tester) any
• func BeforeTest(f Tester) any
• func EchoRenderer() (echo.Renderer, error)
• func RunAsTest(t *testing.T)
• func SaveResult(ctx context.Context, tr TestResult) (string, error)
• func SetDB(db DB)
• func Test(name string, tester Tester) any
• func TestEach[V any](t TestingT, values []V, tester func(TestingT, V))
• type DB
• type InMemoryDB
  • func (db *InMemoryDB) Enumerate(_ context.Context, _ int) (results []Summary, more bool, err error)
  • func (db *InMemoryDB) Load(_ context.Context, id string) (TestResult, error)
  • func (db *InMemoryDB) Save(_ context.Context, result TestResult) (string, error)
• type Level
• type Msg
• type Result
• type Summary
  • func (s Summary) TruncatedTimestamp() time.Time
• type TestResult
  • func LoadResult(ctx context.Context, id string) (TestResult, error)
  • func Run() TestResult
  • func (tr TestResult) FailedSubtests() int
  • func (tr TestResult) FindFailingTests() []TestResult
  • func (tr TestResult) PassedSubtests() int
  • func (tr TestResult) SumTestStats() (total, passed, failed int)
  • func (tr TestResult) TotalSubtests() int
  • func (tr TestResult) TruncatedTimestamp() time.Time
• type Tester
• type TestingT

Variables

var ErrNoDB = errors.New("no DB set")

ErrNoDB indicates no DB has been set via SetDB.

var ErrNotFound = errors.New("not found")

ErrNotFound indicates the provided result ID was not found in the datastore.

Functions

func AddEchoRoutes

func AddEchoRoutes(router *echo.Group)

AddEchoRoutes adds routes to an Echo router that can run tests and retrieve tests results.

func AfterPackage

func AfterPackage(f Tester) any

AfterPackage registers a function to be run once after all tests in the given package have finished. A package may only have one AfterPackage function.

If AfterPackage panics, the reporting behavior depends on if RunAsTest or Run was called. If AfterPackage panics and BeforePackage had already panicked, then the panic for BeforePackage takes priority.

If RunAsTest was called, the panic will be reported as the top-level bootstrap test for the package. If Run was called, every test in the package will be marked as failed, and the panic's message will be appended to every test's own messages or the BeforePackage panic that replaced every test.

When run via Run, any logging output to the provided Tester will only be visible if AfterPackage panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.

NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in AfterPackage to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.

The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:

var _ = testy.AfterPackage(func(){})

func AfterTest

func AfterTest(f Tester) any

AfterTest registers a function to be run once after every top level registered test in the given package has finished. It is not run after subtests created by `t.Run`. A package may only have one AfterTest function.

If AfterTest panics, the specific test that was just run will be marked as failed, and the panic's message will be appended to the test's own messages. Any subtests of the test will not be modified.

When run via Run, any logging output to the provided Tester will only be visible if AfterTest panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.

NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in AfterTest to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.

The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:

var _ = testy.AfterTest(func(){})

func BeforePackage

func BeforePackage(f Tester) any

BeforePackage registers a function to be run once before any tests in the given package are run. A package may only have one BeforePackage function.

If BeforePackage panics, no tests in the package will be run and the reporting behavior depends on if RunAsTest or Run was called. AfterPackage will still be run.

If RunAsTest was called, the panic will be reported as the top-level bootstrap test for the package. If Run was called, every registered test in the package will be marked as failed with the panic's message.

When run via Run, any logging output to the provided Tester will only be visible if BeforePackage panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.

NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in BeforePackage to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.

The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:

var _ = testy.BeforePackage(func(){})

func BeforeTest

func BeforeTest(f Tester) any

BeforeTest registers a function to be run before every top level registered test in the given package is run. It is not run before subtests created by `t.Run`. A package may only have one BeforeTest function.

If BeforeTest panics, the specific registered test that was about to be invoked will be marked as failed with the panic's message. AfterTest will still be run.

When run via Run, any logging output to the provided Tester will only be visible if BeforeTest panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.

NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in BeforeTest to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.

The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:

var _ = testy.BeforeTest(func(){})

func EchoRenderer

func EchoRenderer() (echo.Renderer, error)

EchoRenderer loads the HTML templates and returns an echo.Renderer for the routes provided by this package. Assign this to the Renderer field of your Echo app (or wrap it with your own)

func RunAsTest

func RunAsTest(t *testing.T)

RunAsTest runs all registered tests under Go's testing framework.

To run tests on a per-package basis, put a test file in each package containing a single test that calls this function. This is recommended so accurate per-package execution times are reported, as well as using the test cache. Do not import a test package into another test package as that will cause the tests in the second package to get executed with the first package. If code or resources need shared between test packages, put them in their own package which does not contain any test definitions.

Individual tests in a package may still be run using the standard -run test flag. See `go help testflag` for more information.

TODO: shuffle test execution order (see -shuffle in `go help testflag`)

func SaveResult

func SaveResult(ctx context.Context, tr TestResult) (string, error)

SaveResult saves the provided result to the registered datastore. If no datastore has been registered, an error wrapping ErrNoDB is returned.

func SetDB

func SetDB(db DB)

SetDB sets the datastore to use for test reports. This must be called during application startup.

func Test

func Test(name string, tester Tester) any

Test registers a new test to be run. Tests are run in lexicographical order within a package.

Test cases should *not* be defined in `_test.go` files if they are to be run via Run. If they are, they will not be compiled into the binary. This also means that you need to ensure that your test packages are eventually imported by your main package. You may need to do this with a side effects import (`import _ "my/package"`).

The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:

var _ = testy.Test("my test", func(t testy.TestingT){})

func TestEach

func TestEach[V any](t TestingT, values []V, tester func(TestingT, V))

TestEach runs tester as a subtest for each value in values. The values should have a good default string representation so the subtest names are legible. Consider implementing fmt.Stringer for complex structs.

Types

type DB

type DB interface {
	// Enumerate lists the test results for the given page. The datastore determines the page size.
	Enumerate(ctx context.Context, page int) (results []Summary, more bool, err error)
	// Load retrieves the specified test result from the datastore.
	// If the ID is invalid, ErrNotFound should be returned.
	Load(ctx context.Context, id string) (TestResult, error)
	// Save stores the provided TestResult in the data store and returns its unique ID.
	Save(context.Context, TestResult) (string, error)
}

DB is the interface for something which can save and retrieve test reports.

type InMemoryDB

type InMemoryDB struct {
	// contains filtered or unexported fields
}

InMemoryDB is an implementation of DB that is stored in memory, with no persistent storage. It should be used for demonstration purposes only.

func (*InMemoryDB) Enumerate

func (db *InMemoryDB) Enumerate(_ context.Context, _ int) (results []Summary, more bool, err error)

func (*InMemoryDB) Load

func (db *InMemoryDB) Load(_ context.Context, id string) (TestResult, error)

func (*InMemoryDB) Save

func (db *InMemoryDB) Save(_ context.Context, result TestResult) (string, error)

type Level

type Level string

Level indicates at what log level a Msg was emitted.

const (
	// LevelInfo is an informative log message (Log, etc.)
	LevelInfo Level = "info"
	// LevelError is an error log message (Fatal, etc.)
	LevelError Level = "error"
)

type Msg

type Msg struct {
	Msg   string
	Level Level
}

type Result

type Result string

Result indicates the result of a test.

const (
	// ResultPassed indicates that this test (and all of its subtests) passed.
	ResultPassed Result = "passed"
	// ResultFailed indicates that this test or at least one of its subtests failed.
	ResultFailed Result = "failed"
)

type Summary

type Summary struct {
	// ID is an opaque unique identifier for a test result. The specific format is defined by the datastore.
	ID string
	// Started indicates when a test run was started.
	Started time.Time
	// Dur is how long the test run took to complete.
	Dur time.Duration
	// Total is the total number of tests that were run.
	Total int
	// Passed is the number of tests that passed.
	Passed int
	// Failed is the number of tests that failed.
	Failed int
}

Summary is an overview of a TestResult, used to populate the list of past results.

func (Summary) TruncatedTimestamp

func (s Summary) TruncatedTimestamp() time.Time

TruncatedTimestamp returns the started timestamp truncated to second precision.

type TestResult

type TestResult struct {
	// Package is the Go package that contains the test.
	Package string
	// Name is the name of the test as provided to Test (for top-level tests), Run (for subtests),
	// or the string representation of each value (for TestEach).
	Name string
	// Msgs contains each message that was emitted during the test via the methods on TestingT that emit messages.
	Msgs []Msg
	// Result is the result of the test.
	Result Result
	// Started is when the test was started.
	Started time.Time
	// Dur is how long the test took.
	Dur time.Duration
	// DurHuman is how long the test took in human-readable form.
	DurHuman string
	// Subtests contains the test result of every test this test started via Run or TestEach.
	Subtests []TestResult
}

TestResult is the result of a specific test.

func LoadResult

func LoadResult(ctx context.Context, id string) (TestResult, error)

LoadResult loads the specified result from the registered datastore. If no datastore has been registered, an error wrapping ErrNoDB is returned. If the ID is invalid, an error wrapping ErrNotFound is returned.

func Run

func Run() TestResult

Run runs all registered tests and returns result information about them.

TODO: ability to filter for specific packages and tests

TODO: shuffle test execution order (see -shuffle in `go help testflag`)

TODO: channel for results to support progressive result loading?

func (TestResult) FailedSubtests

func (tr TestResult) FailedSubtests() int

FailedSubtests returns the number of leaf subtests that failed. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.

func (TestResult) FindFailingTests

func (tr TestResult) FindFailingTests() []TestResult

FindFailingTests finds the least deeply nested subtests that have sibling tests that passed. These subtests may be in different branches of subtests. This implies that this test failed; if it did not, then a nil slice is returned. If every subtest of test failed or if test has no subtests, then test itself is returned.

func (TestResult) PassedSubtests

func (tr TestResult) PassedSubtests() int

PassedSubtests returns the number of leaf subtests that passed. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.

func (TestResult) SumTestStats

func (tr TestResult) SumTestStats() (total, passed, failed int)

SumTestStats returns the total number of leaf subtests, as well as the number of those that passed and failed.

func (TestResult) TotalSubtests

func (tr TestResult) TotalSubtests() int

TotalSubtests returns the total number of leaf subtests. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.

func (TestResult) TruncatedTimestamp

func (tr TestResult) TruncatedTimestamp() time.Time

TruncatedTimestamp returns the started timestamp truncated to second precision.

type Tester

type Tester func(t TestingT)

Tester is a thing that runs a test.

type TestingT

type TestingT interface {
	// Fail marks the function as having failed but continues execution.
	Fail()
	// FailNow marks the function as having failed and stops its execution
	// by calling runtime.Goexit (which then runs all deferred calls in the
	// current goroutine).
	// Execution will continue at the next test or benchmark.
	// FailNow must be called from the goroutine running the
	// test or benchmark function, not from other goroutines
	// created during the test. Calling FailNow does not stop
	// those other goroutines.
	FailNow()
	// Fatal is equivalent to Log followed by FailNow.
	Fatal(args ...interface{})
	// Fatalf is equivalent to Logf followed by FailNow.
	Fatalf(format string, args ...interface{})
	// Errorf is equivalent to Logf followed by Fail.
	Errorf(format string, args ...interface{})
	// Helper does not do anything useful since the call stack when passed to the actual implementation has an extra
	// level in it.
	Helper()
	// Log formats its arguments using default formatting, analogous to Println,
	// and records the text in the error log. For tests, the text will be printed only if
	// the test fails or the -test.v flag is set.
	Log(args ...interface{})
	// Logf formats its arguments according to the format, analogous to Printf, and
	// records the text in the error log. A final newline is added if not provided. For
	// tests, the text will be printed only if the test fails or the -test.v flag is
	// set.
	Logf(format string, args ...interface{})
	// Run runs f as a subtest of t called name. It runs f in a separate goroutine
	// and blocks until f returns (or, if running via go test, calls t.Parallel to become a parallel test).
	// Run reports whether f succeeded (or, if running via go test, at least did not fail before calling t.Parallel).
	//
	// Run may be called simultaneously from multiple goroutines, but all such calls
	// must return before the outer test function for t returns.
	Run(string, Tester) bool
	// Parallel signals that this test is to be run in parallel with (and only with)
	// other parallel tests. When a test is run multiple times due to use of
	// -test.count or -test.cpu, multiple instances of a single test never run in
	// parallel with each other.
	//
	// Parallel only affects RunAsTest as it relies on testing.T's implementation.
	Parallel()
}

TestingT is a subset of testing.T that we have to implement for non-`go test` runs.

TODO flesh this out with more useful stuff from testing.T -- Parallel would be nice but tricky