check

README

Checking...

Intro

Go Check is a property based testing (also known as generative testing) framework for Go programming language. In property based testing, you state facts about your code that for given precondition should always be true. Framework then tests these facts with random inputs to verify the statements. If the statement doesn't hold, framework tries to find the input with smallest complexity for which it doesn't.

Motivation

Writting tests gives a confidence that written code works, and rewritten code doesn't break. One of the standard ways of writting tests is by defining test cases, a set of inputs for which it needs to be ensured that code under test works.

With property based testing paradigm of writting tests is shifted. Test cases are generated by the framework and writting tests comes down to defining properties that should always hold.

Installation

Installation is done as any other go package:

go get github.com/steffnova/go-check

Quick Start

Go Check works seamlesly with testing package. When writting tests with go-check user doesn't specify test input but input generators. These generators are then then used, through multple iterations (100 by default), to generate inputs and feed them to test. The example below (commutative property for addition) describes key components for defining go-check tests:

package main_test

import (
    "fmt"
    "testing"

    "github.com/steffnova/go-check"
    "github.com/steffnova/go-check/generator"
    "github.com/steffnova/go-check/property"
)



func TestAdditionCommutativity(t *testing.T) {
    // Predicate is a function that will be used for defining a property. Predicate
    // can have variable number of inputs, but must have only one output of error type
    predicate := property.Predicate(func(x, y int) error {
        // Test if changing the order of operands in addition gives the same result
        if x+y != y+x {
            return fmt.Errorf("commutativity doesn't hold for addition")
        }
        return nil
    })

    // Predicate function has two inputs of type int, because of that
    // 2 generators are required (one for "x" and one for "y"). Generators
    // must generate int values.
    // generator.Int accepts constraints that define range of generated values.
    // If no constraints are specified, any int value can be generated
    inputs := property.Inputs(
        generator.Int(),
        generator.Int(),                               
    )

    // Property requires a predicate function and input generators. 
    // Number of generators must match number of inputs predicate has.
    // Generator's type with index i must match predicate's input with index i.
    property := property.Define(
        inputs,
        predicate,    
    )

    // Check tests the property by feeding it with generated values
    check.Check(t, property)
}

Commutativity is one of defining properties for addition, and test above will always pass. To demonstrate shrinking capabilities of go-check, following example will test commutativity for subtraction

package main

import (
	"fmt"
	"testing"

	"github.com/steffnova/go-check"
	"github.com/steffnova/go-check/generator"
	"github.com/steffnova/go-check/property"
)

func TestSubtractionCommutativity(t *testing.T) {
	check.Check(t, property.Define(
		property.Inputs(
			generator.Int(),
			generator.Int(),
		),
		property.Predicate(func(x, y int) error {
			if x-y != y-x {
				return fmt.Errorf("commutativity does not hold for subtraction. ")
			}
			return nil
		}),
	))
}

Commutativity is not property of subtraction and thus above example will always fail. There are many combination of x and y for which test fails, but understing why it fails is not always visible at the first glance, especially if x and y are large values. Most of the generators also support shrinking. Shrinking is a process of reducing complexity of test inputs, and will be executed on first failing value. For this example, this means reducing the int values of x and y to smaller values, while making sure that test is still failing. Once the shrinking process is done it will present test results:

--- FAIL: TestSubtractionCommutativity (0.00s)
    Check failed after 1 tests with seed: 1646421732271105000. 
    Property failed for inputs: [
        <int> 0,
        <int> 1
    ]
    Shrunk 123 time(s)
    Failure reason: commutativity does not hold for subtraction.  
    
    Re-run:
    go test -run=TestSubtractionCommutativity -seed=1646421732271105000 -iterations=100

Test result display the number of test ran before test failed, seed that was used to feed random number generation, smallest possible set of values for which test fails, number of times shrinking occured and failing error message. It is very important to be able to reproduce the failing test and for that reason command that can be used to reproduce test failure is printed at the end.

go-check accept two flag parameters that can be added to go test command:

• seed, seed for random number generator used by all generators
• iterations, total number of test go-check will perform

Documentation

• Generators

Documentation

Overview

* Check and it's sub packages provide tool for writing property based tests

Index

• func Check(t *testing.T, property property.Property, config ...Config)
• type Config

Functions

func Check

func Check(t *testing.T, property property.Property, config ...Config)

Check checks if property holds. First parameter is *testing.T that will report error if property returns an error (property doesn't hold). The property paramter should be defined using property.Define. The config parameter, even though it is a variadice parameter it uses only the first instance of Config passed to it. If config is not specified default configuration is used (random seed and 100 iterations). Config will run property number of times equal to Iterations values, specified by config parameter or (100 if default value is used). Following example demonstrates how to use Check in tests:

package main_test

import (
    "fmt"
    "testing"

    "github.com/steffnova/go-check"
    "github.com/steffnova/go-check/generator"
    "github.com/steffnova/go-check/property"
)

func TestSubtractionCommutativity(t *testing.T) {
    check.Check(t, property.Define(
        property.Inputs(
            generator.Int()
            generator.Int()
        )
        property.Predicate(func(x, y int) error {
            if x-y != y-x {
                return fmt.Errorf("commutativity does not hold for subtraction.")
            }
            return nil
        })
    ))
}

Commutativity does not hold for subtraction operation for every x and y as it is defined in predicate. Because of this an error will be thrown that will report value for which property failed. If shrinking is possible, property.Define will report smallest possible value for which predicate didn't hold.

--- FAIL: TestSubtractionCommutativity (0.00s)
    Check failed after 1 tests with seed: 1646421732271105000.
    Property failed for inputs: [
        <int> 0,
        <int> 1
    ]
    Shrunk 123 time(s)
    Failure reason: commutativity does not hold for subtraction.

    Re-run:
    go test -run=TestSubtractionCommutativity -seed=1646421732271105000 -iterations=100

Types

type Config

type Config struct {
	Seed       int64 // Seed used by random number generator
	Iterations int64 // Number of times property will be checked
}

Config is configuration used by Check.