subtest

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 28, 2020 License: Apache-2.0 Imports: 12 Imported by: 0

README

subtest

Go Report Card GoDev

subtest is a minimalist Go test-utility package used to initializing small test functions for use with the Go sub-tests feature. You can read more about Go sub-tests here.

The sub-package subjson defines middleware for parsing values from JSON before performing checks.

Introduction

subtest was motivated by a desire to make it easier write Given-When-Then (GWT) style tests in Go on top of the built-in test-runner, and without a DSL.

GWT is a naming schema for tests that attempts to clarify three aspects:

  1. Given: How does the world look like before we do an action.
  2. When: What do we do to affect the world.
  3. Then: How should the world look like after the action.

These conditions can be nested to test different scenarios. Here is an example:

Given foo is 42
    When dividing by 6
        Then the result should be 7
    When dividing by 9
        Then the result should be less than 5
        Then the result should be more than 4

A common believe is that to write GWT style tests in Go, you should use a Behavior-Driven-Development framework and associated Domain-Specific-Language. However, we will argue that this simply isn't true.

One of the problems with many Behavior-Driven-Development frameworks in Go, is that they tend to rely on their own test-runner and sub-test logic. While there could be good reasons for using this, it also comes with a price: tooling expecting the default test runner simply does not cope. This is true either we are talking about a CI that fail to parse sub-test results into JUnit summaries, an IDE that fail to insert links for navigating to the failing code, or a command-line tool for rerunning failing sub-tests by name. This has a real impact on how well test-results can be understood when used with tooling.

As it turns out, you can actually write GWT-style tests not only without a BDD framework or DSL, but without a framework or library what so ever:

func TestFoo(t *testing.T) {
    t.Run("Given foo is 42", func(t *testing.T) {
        const foo = 42
        t.Run("When dividing by 6", func (t *testing.T) {
            v := float64(foo) / 6
            t.Run("Then the result should be 7", func(t *testing.T) {
                if expect := float64(7); v != expect {
                    t.Fatalf("\n got: %d\nwant: %d", v, expect)
                }
            })
        })
        t.Run("When dividing by 9", func (t *testing.T) {
            v := float64(foo) / 9
            t.Run("Then the result should be greater than 4" func(t *testing.T) {
                if expect := float64(4); v > expect {
                    t.Fatalf("\n got: %d\nwant > %d", v, expect)
                }
            })
            t.Run("Then the result should be less than 5" func(t *testing.T) {
                if expect := float64(5); v < expect {
                    t.Fatalf("\n got: %d\nwant < %d", v, expect)
                }
            })
        })
    })
}

While doing this is fine, it can quickly become repetitive. It can also become challenging to maintain consistent output for failing tests over time; in particularly so for a growing team.

By now you might think that subtest is going to improve how to write GWT style tests without a BDD-style framework, and you are right. However, there is nothing within the design of subtest that restricts it to handling GWT style tests. Instead, subtest is a generalized test utility package for generating sub-tests, no matter your style preferences. In other words, you can use subtest to write GWT style tests, table-driven tests, or some other style that you prefer.

Here is a version of TestFoo with subtest:

func TestFoo(t *testing.T) {
    t.Run("Given foo is 42", func(t *testing.T) {
        const foo = 42
        t.Run("When dividing by 6", func (t *testing.T) {
            vf := subtest.Value(float64(foo) / 6)
            t.Run("Then the result should be 7", vf.NumericEqual(7))
        })
        t.Run("When dividing by 9", func (t *testing.T) {
            vf := subtest.Value(float64(foo) / 6)
            t.Run("Then the result should be greater than 4", vf.GreaterThan(4))
            t.Run("Then the result should be less than 5", vf.LessThan(5))
        })
    })
}

Usage

Illustrative example

Building and running a subtest is generally composed of six steps. Normally you do not do each steps as explicit as described below, but to illustrate the general flow, we have spelled this out.

 // TestFooExplicit shows the different steps of building a subtest.
func TestFooExplicit(t *testing.T) {
    // 1. We declare the value we want to check; usually the result of an
    // operation or action.
    v := "foo"

    // 2. We initialize a value function for the value that we want to check.
    // There are several different initializers we can call to get a value
    // function; this is the simplest one.
    vf := subtest.Value(v)

    // 3. We initialize the check we want to use. A check is anything that
    // implements the Check interface.
    c := subtest.NumericEquals(3)

    // 4. We can optionally wrap our check with middleware.
    c = subtest.OnLen(c)

    // 5. We initialize a test function by passing the check to the value
    // function's Test method.
    tf := vf.Test(c)

    // 6. We run the test function as a sub-test.
    t.Run("len(v) == 3", tf)
}

If we where going to do this every time, we would grow weary. Therefore there is several short-hand methods defined on the Check and ValueFunc instances that makes things easier. The least verbose variant we can write of the test above is as follows:

func TestFoo(t *testing.T) {
    v := "foo"

    t.Run("len(v) == 3", subtest.Len(v).NumericEquals(3))
}

JSON Schema validation

It is possible to validate more than just equality with subtest. The subtest.Schema type allows advanced validation of any Go map type, and in the future, perhaps also for structs. From the subjson package we can use ValueFunc initializers, Check implementations and check middleware to decode JSON from string, []byte and json.RawMessage values. Combining these two mechanisms we can do advanced validation of JSON content.

func TestJSONMap(t *testing.T) {
    v := `{"foo": "bar", "bar": "foobar", "baz": ["foo", "bar", "baz"]}`

    expect := subtest.Schema{
        Fields: subtest.Fields{
            "foo": subjson.DecodesTo("bar")
            "bar": subjson.OnLen(subtest.AllOf{
                subtest.GreaterThan(3),
                subtest.LessThan(8),
            }),
            "baz": subjson.OnSlice(subtest.AllOf{
                subtest.OnLen(subtest.DeepEqual(3)),
                subtest.OnIndex(0, subjson.DecodesTo("foo"),
                subtest.OnIndex(1, subtest.MatchPattern(`"^b??$"`), // regex match against raw JSON
                subtest.OnIndex(2, subtest.DeepEqual(json.RawMessage(`"baz"`)), // raw JSON equals
            }),
        },
    }

    t.Run("match expectations", subjson.Map(v).Test(expect))
}
Required checks

This is perhaps not commonly known, but the t.Run function actually return false if there is a failure. Or to be more accurate:

Run reports whether f succeeded (or at least did not fail before calling t.Parallel).

Because subtest checks do not call t.Parallel, this can be utilized to stop test-execution if a "required" sub-test fails.

func TestFoo(t *testing.T) {
    v, err := foo()

    if !t.Run("err == nil", subtest.Value(err).NoError()) {
        // Abort further tests if failed.
        t.FailNow()
    }
    // Never run when err != nil.
    t.Run("v == foo", subtest.Value(v).DeepEqual("foo"))
}

func foo() (string, error) {
    return "", errors.New("failed")
}
Extendability

The subtest library itself is currently zero-dependencies. The important aspect of this is that we do not force opinionated dependencies on the user. However, it's also written to be relatively easy to extend.

For specialized use cases and customization, see the examples/ sub-directory:

  • examples/gwt: Example of tests following the Given-When-Then naming convention.
  • examples/colorfmt: Example of custom type formatting with colors via the pp package.
  • examples/gojsonq: Example of custom checks for JSON matching via the gojsonq package.
  • examples/jsondiff: Example of custom checks for JSON comparison via the jsondiff package.

Features

Some key features of subtest is described below.

Utilize the standard test runner

subtest initializes test functions intended for usage with the Run method on the testing.T type, and uses a plain output format by default. This means that tooling and IDE features built up around output from the standard test runner will work as expected.

Check State-ful values

Values to check are wrapped in a value function (ValueFunc). By setting up your own value function, you can easily run several tests against state-ful types, such as an io.Reader, where each check starts with a clean slate.

Check middleware

Generally, a sub-test performs of a single check (CheckFunc). These checks can be wrapped by middleware to facilitate processing or transformation of values before running nested checks. E.g. parse a byte array from JSON into a Go type, or extract the length of an array.

Plain output

The quicker a failed test can be understood, the quicker it can be fixed. subtest's default failure formatting is inspired by the short and simplistic style used for unit tests within the Go standard library. We have extended this syntax only so that we can more easily format the expected type and value.

Example output from an exaples/gwt:

--- FAIL: TestFoo (0.00s)
    --- FAIL: TestFoo/Given_nothing_is_registered (0.00s)
        --- FAIL: TestFoo/Given_nothing_is_registered/When_calling_reg.Foo (0.00s)
            --- FAIL: TestFoo/Given_nothing_is_registered/When_calling_reg.Foo/Then_the_result_should_hold_a_zero-value (0.00s)
                pkg_test.go:19: not deep equal
                    got: string
                        "oops"
                    want: string
                        ""
FAIL
FAIL	github.com/searis/subtest/examples/gwt	0.057s
FAIL

Be aware that the default type formatter currently do not expand nested pointer values.

Custom formatting

While we aim to make the default type formatting useful, it will also be somewhat limited due to our zero-dependency goal. Type formatting is also an area with potential for different opinions on what looks the most clear. For this reason we have made it easy to replace the default type formatter using libraries such as go-spew, litter, or pp (with colors).

Example using go-spew:

import (
    "github.com/davecgh/go-spew/spew"
    "github.com/searis/subtest"
)

func init() {
    subtest.SetTypeFormatter(spew.ConfigState{Indent: "\t"}.Sdump)
}

Example using litter:

import (
    "github.com/sanity-io/litter"
    "github.com/searis/subtest"
)

func init() {
    subtest.SetTypeFormatter(litter.Options{}.Sdump)
}

Example using pp with conditional coloring:

import (
    "golang.org/x/crypto/ssh/terminal"
    "github.com/k0kubun/pp"
    "github.com/searis/subtest"
)

func init() {
    subtest.SetTypeFormatter(pp.Sprint)

    colorEnv := strings.ToUpper(os.Getenv("GO_TEST_COLOR"))
    switch colorEnv {
    case "0", "FALSE":
      log.Println("explicitly disabling color output for test")
        pp.ColoringEnabled = false
    case "1", "TRUE":
        log.Println("explicitly enabling color output for test")
    default:
        if !terminal.IsTerminal(int(os.Stdout.Fd())) {
            log.Println("TTY not detected, disabling color output for test")
            pp.ColoringEnabled = false
        } else {
            log.Println("TTY detected, enabling color output for test")
        }
    }
}

When it comes to prettifying the output of the test runner itself, there are separate tools for that. One such tool is gotestsum, which wraps the Go test runner to provide alternate formatting.

Documentation

Overview

Package subtest provides a way of intializing small test functions suitable for use as with the (*testing.T).Run method. Tests using package-defined check functions can generally be initialized in two main ways. here is an example using DeepEquals:

// Short-hand syntax for built-in check functions.
t.Run("got==expect", subtest.Value(got).DeepEquals(expect))

// Long syntax.
 t.Run("got==expect", subtest.Value(got).Test(subtest.DeepEquals(expect)))

Custom CheckFunc implementations can also be turned into tests:

t.Run("got==expect", subtest.Value(got).Test(func(got interface{}) error {
    if got != expect {
        return subtest.FailExpect("not equal", got, expect)
    }
}))

Experimentally, any function that takes no parameter and returns an error can also be converted to a test:

t.Run("got==expect", subtest.Test(func() error {
    if got != expect {
        return subtest.FailExpect("not plain equal", got, expect)
    }
}))

When necessary, either CheckFunc middleware or a custom ValueFunc instances can be used to prepare or transform the test value before evaluation.

PS! Note that the all experimental syntax may be removed in a later version.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FormatType

func FormatType(v interface{}) string

FormatType formats a type using the configured type formatter for the package.

func KeyError

func KeyError(key interface{}, err error) error

KeyError returns an error prefixed by a key.

func SetIndent

func SetIndent(s string)

SetIndent sets a string to use in package error and type formatting. The default is four spaces, as that's what's used by the Go test-runner. This function is not thread-safe, and should be called as part of initialization only. E.g. in a test package init function.

func SetTypeFormatter

func SetTypeFormatter(f func(...interface{}) string)

SetTypeFormatter replaces the type formatter used by the package. This function is not thread-safe, and should be called as part of initialization only. E.g. in a test package init function.

func Test

func Test(f func() error) func(t *testing.T)

Test returns a test that fails fatally with the error f returned by f.

Types

type AllOff

type AllOff []Check

AllOff is a Check type that fails if any of it's members fails.

func (AllOff) Check

func (cs AllOff) Check(vf ValueFunc) error

Check runs all member checks and returns an aggregated error of at least one check fails.

type Check

type Check interface {
	Check(vf ValueFunc) error
}

Check describes the interface for a check.

type CheckFunc

type CheckFunc func(got interface{}) error

CheckFunc is a function that return an error on failure.

func Any

func Any() CheckFunc

Any returns a no-operation check function that never fails.

func Before

func Before(expect time.Time) CheckFunc

Before returns a check function that fails when the test value is not before expect. Accepts type time.Time and *time.Time.

func Contains

func Contains(v interface{}) CheckFunc

Contains returns a check function that fails if the test value does not contain the input. Accepts input of type array and slice.

func ContainsMatch

func ContainsMatch(c Check) CheckFunc

ContainsMatch returns a check function that fails if the test value does not contain the check. Accepts input of type array and slice.

func DeepEqual

func DeepEqual(expect interface{}) CheckFunc

DeepEqual returns a check function that fails when the test value does not deep equals to expect.

func Error

func Error() CheckFunc

Error returns a check function that fails if the test value is nil or not an error type.

func ErrorIs

func ErrorIs(target error) CheckFunc

ErrorIs returns a check function that fails if the test value is not an error matching target, or not an error type.

func ErrorIsNot

func ErrorIsNot(target error) CheckFunc

ErrorIsNot returns a check function that fails if the test value is an error matching target, or not an error type.

func GreaterThan

func GreaterThan(expect float64) CheckFunc

GreaterThan returns a check function that fails when the test value is not a numeric value greater than expect.

func GreaterThanOrEqual

func GreaterThanOrEqual(expect float64) CheckFunc

GreaterThanOrEqual returns a check function that fails when the test value is not a numeric value greater than or equal to expect.

func LessThan

func LessThan(expect float64) CheckFunc

LessThan returns a check function that fails when the test value is not a numeric value less than expect.

func LessThanOrEqual

func LessThanOrEqual(expect float64) CheckFunc

LessThanOrEqual returns a check function that fails when the test value is not a numeric value less than or equal to expect.

func MatchPattern

func MatchPattern(pattern string) CheckFunc

MatchPattern is a short-hand for MatchRegexp(regexp.MustCompile(pattern)).

func MatchRegexp

func MatchRegexp(r *regexp.Regexp) CheckFunc

MatchRegexp returns a check function that fails if the test value does not match r. Allowed test value types are string, []byte, json.RawMessage, io.RuneReader and error.

func NoError

func NoError() CheckFunc

NoError returns a check function that fails when the test value is a non-nill error, or if it's not an error type.

func NotBefore

func NotBefore(expect time.Time) CheckFunc

NotBefore returns a check function that fails when the test value is before expect. Accepts type time.Time and *time.Time.

func NotDeepEqual

func NotDeepEqual(reject interface{}) CheckFunc

NotDeepEqual returns a check function that fails when the test value deep equals to reject.

func NotNumericEqual

func NotNumericEqual(expect float64) CheckFunc

NotNumericEqual returns a check function that fails when the test value is a numeric value equal to expect.

func NotReflectNil

func NotReflectNil() CheckFunc

NotReflectNil returns a check function that fails when the test value is either an untyped nil value or reflects to a pointer with a nil value.

func NotTimeEqual

func NotTimeEqual(expect time.Time) CheckFunc

NotTimeEqual returns a check function that fails when the test value is a time semantically equal to expect. Accepts type time.Time and *time.Time.

func NumericEqual

func NumericEqual(expect float64) CheckFunc

NumericEqual returns a check function that fails when the test value is not a numeric value equal to expect.

func OnCap

func OnCap(c Check) CheckFunc

OnCap returns a check function where the capacity of the test value is extracted and passed to c. Accepted input types are arrays, slices and channels.

func OnFloat64

func OnFloat64(c Check) CheckFunc

OnFloat64 returns a check function where the test value is converted to float64 before it's passed to c.

func OnIndex

func OnIndex(i int, c Check) CheckFunc

OnIndex returns a check function where the item at index i of the test value is passed on to c. Accepted input types are arrays, slices and strings.

func OnLen

func OnLen(c Check) CheckFunc

OnLen returns a check function where the length of the test value is extracted and passed to c. Accepted input types are arrays, slices, maps, channels and strings.

func ReflectNil

func ReflectNil() CheckFunc

ReflectNil returns a check function that fails when the test value is neither an untyped nil value nor reflects to a pointer with a nil value.

func TimeEqual

func TimeEqual(expect time.Time) CheckFunc

TimeEqual returns a check function that fails when the test value is not a time semantically equal to expect. Accepts type time.Time and *time.Time.

func (CheckFunc) Check

func (f CheckFunc) Check(vf ValueFunc) error

Check runs the check function against a value function.

type Errors

type Errors []error

Errors combine the output of multiple errors on separate lines.

func (Errors) Error

func (errs Errors) Error() string

func (Errors) Is

func (errs Errors) Is(target error) bool

Is returns true if target is found within errs or if target deep equals errs.

type Failure

type Failure struct {
	Prefix string
	Got    string
	Expect string
	Reject string
	// contains filtered or unexported fields
}

Failure is an error type that aid with consistent formatting of test failures. In error matching, two Failure instances are considered equal when their formattet content is the same.

func FailExpect

func FailExpect(prefix string, got, expect interface{}) Failure

FailExpect formats a failure for content that is not matching some expected value. The package type formatter is used.

func FailGot

func FailGot(prefix string, got interface{}) Failure

FailGot formats a failure for some unexpected content. The package type formatter is used.

func FailReject

func FailReject(prefix string, got, reject interface{}) Failure

FailReject formats a failure for content that is matching some rejected value. The package type formatter is used.

func Failf

func Failf(format string, v ...interface{}) Failure

Failf formats a plain text failure.

func (Failure) Error

func (f Failure) Error() string

func (Failure) Is

func (f Failure) Is(target error) bool

Is returns true if f matches target.

func (Failure) Unwrap

func (f Failure) Unwrap() error

Unwrap returns the next error in the chain, if any.

type Fields

type Fields map[interface{}]Check

Fields provides a map of check functions.

func (Fields) OrderedKeys

func (m Fields) OrderedKeys() []interface{}

OrderedKeys returns all keys in m in alphanumerical order.

type Schema

type Schema struct {
	// Fields map keys to checks.
	Fields Fields
	// Required, if set, contain a list of required keys. When the list is
	// explicitly defined as an empty list, no keys will be considered required.
	// When the field holds a nil value, all keys present in Fields will be
	// considered required.
	Required []interface{}
	// AdditionalFields if set, contain a check used to validate all fields
	// where the key is not present in Fields. When the field holds a nil
	// value, no additional keys are allowed. To skip validation of additional
	// keys, the Any() check can be used.
	AdditionalFields Check
}

Schema allows simple validation of fields. Currently support only maps.

func (Schema) Check

func (s Schema) Check(vf ValueFunc) error

Check validates vf against s. For now, vf must return a map.

type ValueFunc

type ValueFunc func() (interface{}, error)

ValueFunc is a function returning a value. The main purpose of a ValueFunc instance is to initialize tests against the the returned value.

func Cap

func Cap(v interface{}) ValueFunc

Cap returns a new ValueFunc for the capacity of v.

func Float64

func Float64(v interface{}) ValueFunc

Float64 returns a new ValueFunc that parses v into a float64. Accepts numeric kinds and string kinds as input.

func Index

func Index(v interface{}, i int) ValueFunc

Index returns a new ValueFunc for the value at index v[i]. Accepts input of type array, slice and string.

func Len

func Len(v interface{}) ValueFunc

Len returns a new ValueFunc for the length of v.

func Value

func Value(v interface{}) ValueFunc

Value returns a new ValueFunc for a static value v.

func (ValueFunc) Before

func (vf ValueFunc) Before(v time.Time) func(t *testing.T)

Before is equivalent to vf.Test(Before(v)).

func (ValueFunc) Contains

func (vf ValueFunc) Contains(v interface{}) func(t *testing.T)

Contains is equivalent to vf.Test(Contains{v}).

func (ValueFunc) ContainsMatch

func (vf ValueFunc) ContainsMatch(c Check) func(t *testing.T)

ContainsMatch is equivalent to vf.Test(ContainsMatch{c}).

func (ValueFunc) DeepEqual

func (vf ValueFunc) DeepEqual(v interface{}) func(t *testing.T)

DeepEqual is equivalent to vf.Test(DeepEqual(v)).

func (ValueFunc) Error

func (vf ValueFunc) Error() func(t *testing.T)

Error is equivalent to vf.Test(Error(v)).

func (ValueFunc) ErrorIs

func (vf ValueFunc) ErrorIs(target error) func(t *testing.T)

ErrorIs is equivalent to vf.Test(ErrorIs(v)).

func (ValueFunc) ErrorIsNot

func (vf ValueFunc) ErrorIsNot(target error) func(t *testing.T)

ErrorIsNot is equivalent to vf.Test(ErrorIsNot(v)). wrapped by vf.

func (ValueFunc) GreaterThan

func (vf ValueFunc) GreaterThan(v float64) func(t *testing.T)

GreaterThan is equivalent to vf.Test(GreaterThan(v)).

func (ValueFunc) GreaterThanOrEqual

func (vf ValueFunc) GreaterThanOrEqual(v float64) func(t *testing.T)

GreaterThanOrEqual is equivalent to vf.Test(GreaterThanOrEqual(v)).

func (ValueFunc) LessThan

func (vf ValueFunc) LessThan(v float64) func(t *testing.T)

LessThan is equivalent to vf.Test(LessThan(v)).

func (ValueFunc) LessThanOrEqual

func (vf ValueFunc) LessThanOrEqual(v float64) func(t *testing.T)

LessThanOrEqual is equivalent to vf.Test(LessThanOrEqual(v)).

func (ValueFunc) MatchPattern

func (vf ValueFunc) MatchPattern(pattern string) func(t *testing.T)

MatchPattern is equivalent to vf.Test(s.MatchPattern(pattern)).

func (ValueFunc) MatchRegexp

func (vf ValueFunc) MatchRegexp(r *regexp.Regexp) func(t *testing.T)

MatchRegexp is equivalent to vf.Test(s.MatchRegexp(r)).

func (ValueFunc) NoError

func (vf ValueFunc) NoError() func(t *testing.T)

NoError is equivalent to vf.Test(NoError(v)).

func (ValueFunc) NotBefore

func (vf ValueFunc) NotBefore(v time.Time) func(t *testing.T)

NotBefore is equivalent to vf.Test(NotBefore(v)).

func (ValueFunc) NotDeepEqual

func (vf ValueFunc) NotDeepEqual(v interface{}) func(t *testing.T)

NotDeepEqual is equivalent to vf.Test(NotDeepEqual(v)).

func (ValueFunc) NotNumericEqual

func (vf ValueFunc) NotNumericEqual(v float64) func(t *testing.T)

NotNumericEqual is equivalent to vf.Test(NotNumericEqual(v)).

func (ValueFunc) NotReflectNil

func (vf ValueFunc) NotReflectNil() func(t *testing.T)

NotReflectNil is equivalent to vf.Test(NotReflectNil(v)).

func (ValueFunc) NotTimeEqual

func (vf ValueFunc) NotTimeEqual(v time.Time) func(t *testing.T)

NotTimeEqual is equivalent to vf.Test(NotTimeEqual(v)).

func (ValueFunc) NumericEqual

func (vf ValueFunc) NumericEqual(v float64) func(t *testing.T)

NumericEqual is equivalent to vf.Test(NumericEqual(v)).

func (ValueFunc) ReflectNil

func (vf ValueFunc) ReflectNil() func(t *testing.T)

ReflectNil is equivalent to vf.Test(ReflectNil(v)).

func (ValueFunc) Test

func (vf ValueFunc) Test(c Check) func(t *testing.T)

Test returns a test function that fails fatally with the error returned by f.Check(vf).

func (ValueFunc) TimeEqual

func (vf ValueFunc) TimeEqual(v time.Time) func(t *testing.T)

TimeEqual is equivalent to vf.Test(TimeEqual(v)).

Directories

Path Synopsis
examples module
internal
Package subjson contains check function middelware that validates and decodes / JSON before passing it on to another check function.
Package subjson contains check function middelware that validates and decodes / JSON before passing it on to another check function.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL