cptest

README

cptest

Copy all example test cases from a problem statement into a file. Then test your code in one command!

Hmm?

Let's assume that you have a shortcut for compiling your code. Also the directory with your code looks like this

$ ls
app Makefile main.cpp

Create a file, say, inputs.txt and copy the test cases in the following format:

input: test 1
---
output for test 1
===
input: test 2
---
output for test 2

That is, just separate input and output with --- and test cases themselves with ===.

Now, simply run in the directory

$ cptest
found executable: /home/username/path/to/app
using default time limit: 6s
=== RUN Test 1
=== RUN Test 2
=== RUN Test 3
--- WA: Test 1 (0.005s)
Input:
input: test 1

Answer:
output for test 1

Output:
no

--- WA: Test 2 (0.006s)
Input:
input: test 2

Answer:
output for test 2

Output:
no

(Your code was determining if the given string was a palindrome, by the way.)

You can also explicitly provide the working directory and/or the path to the inputs/executable. In general

Usage:
    cptest -i INPUTS -e EXECUTABLE [WORKING_DIR]

You can simply amend the command you're executing with the shortcut and that's it! No more risking getting a WA on the first test or wasting time rechecking the given examples!

Time limit

You can also specify a time limit for the tests. Instead of the first test, you can provide a set of key-value pairs. Only one key is supported for now, though -- tl. (If you have any ideas for more, an issue is welcome!)

Assuming, the task is to compute fibonacci numbers, somebody wrote a O(2^n) implementation. This inputs.txt will fail:

tl = 10.0
===
1
---
1
===
2
---
1
===
47
---
2971215073

Running cptest, we get

$ cptest
found executable: /home/username/path/to/app
=== RUN	Test 1
=== RUN	Test 2
=== RUN	Test 3
--- OK:	Test 1 (0.006s)
--- OK:	Test 2 (0.006s)
--- TL:	Test 3 (10.000s)
Input:
47

Answer:
2971215073

FAIL
2/3 passed

Build

To build and run the application, do (assuming you've cloned the repository)

$ cd cptest
$ go build ./cmd/cptest
$ ./cptest

You can add it to your PATH to call it from anywhere in the console. On Linux that could be /usr/bin/ directory.

Documentation

Index

• Constants
• func AssertCallCount(t *testing.T, got, want int)
• func AssertConfig(t *testing.T, got, want map[string]string)
• func AssertErrorLines(t *testing.T, errs []error, lines []int)
• func AssertErrors(t *testing.T, got, want []error)
• func AssertNoConfig(t *testing.T, got map[string]string)
• func AssertNoErrors(t *testing.T, errs []error)
• func AssertTest(t *testing.T, got Test, want Test)
• func AssertTests(t *testing.T, got []Test, want []Test)
• func AssertTimes(t *testing.T, got, want map[int]time.Duration)
• func AssertVerdicts(t *testing.T, got, want map[int]Verdict)
• func BlankResultPrinter(b *TestingBatch, test Test, id int)
• func ScanConfig(text string) (m map[string]string, errs []error)
• func ScanKeyValuePair(line string) (string, string, error)
• func VerboseResultPrinter(b *TestingBatch, test Test, id int)
• type ConfigurableStopwatcher
  • func NewConfigurableStopwatcher(TL time.Duration) *ConfigurableStopwatcher
  • func (s *ConfigurableStopwatcher) Elapsed() time.Duration
  • func (s *ConfigurableStopwatcher) TimeLimit() <-chan time.Duration
• type Executable
  • func (e *Executable) Run(r io.Reader, w io.Writer) error
• type Inputs
  • func ScanInputs(r io.Reader) (inputs Inputs, errs []error)
• type InputsError
  • func (e InputsError) Error() string
• type InternalError
  • func (e InternalError) Error() string
• type LinedError
  • func (e *LinedError) Error() string
  • func (e *LinedError) Unwrap() error
• type PrintResultFunc
• type Processer
• type ProcesserFunc
  • func (p ProcesserFunc) Run(r io.Reader, w io.Writer) error
• type SpyProcesser
  • func (p *SpyProcesser) Run(r io.Reader, w io.Writer) error
• type SpyStopwatcher
  • func (s *SpyStopwatcher) Elapsed() time.Duration
  • func (s *SpyStopwatcher) TimeLimit() <-chan time.Duration
• type Stopwatcher
• type Test
  • func ScanTest(str string) (Test, []error)
• type TestingBatch
  • func NewTestingBatch(inputs Inputs, proc Processer, swatch Stopwatcher) *TestingBatch
  • func (b *TestingBatch) Run()
• type Verdict

Constants

const (
	IOSeparatorMissing = InputsError("IO separator missing")
	KeyMissing         = InputsError("key cannot be empty")
	ValueMissing       = InputsError("value cannot be empty")
	KVMissing          = InputsError("key and value are missing")
	NotKVPair          = InputsError("not a key-value pair")
)

A set of errors that may be produced during scanning of the inputs file. These replace sentinel errors making the errors be comparable with equals operator.

const (
	IODelim   = "---"
	TestDelim = "==="
)

The set of delimeters used when partitioning inputs file.

Functions

func AssertCallCount

func AssertCallCount(t *testing.T, got, want int)

AssertCallCount checks that the received and expected number of calls are equal.

func AssertConfig

func AssertConfig(t *testing.T, got, want map[string]string)

AssertConfig checks whether received and expected config key-value sets are equal.

func AssertErrorLines

func AssertErrorLines(t *testing.T, errs []error, lines []int)

AssertErrorLines checks that each error in the received array of errors is wrapping a LinedError error. At the same time, it checks that the line numbers are equal to the expected ones.

func AssertErrors

func AssertErrors(t *testing.T, got, want []error)

AssertErrors compared received array of errors with the expected one.

func AssertNoConfig

func AssertNoConfig(t *testing.T, got map[string]string)

AssertNoConfig checks that the received key-value set is empty. If it's not, the test is failed and the its contents are printed.

func AssertNoErrors

func AssertNoErrors(t *testing.T, errs []error)

AssertNoErrors will check if the array of errors is empty. If it's not empty, the test will be failed and the errors will be reported.

func AssertTest

func AssertTest(t *testing.T, got Test, want Test)

AssertTest compare the inputs and outputs with respective expected ones for equivalence.

func AssertTests

func AssertTests(t *testing.T, got []Test, want []Test)

AssertTests will compare received array of tests with the expected one.

func AssertTimes

func AssertTimes(t *testing.T, got, want map[int]time.Duration)

AssertTimes check whether the received and expected timestampts for the test cases both exist and are equal.

func AssertVerdicts

func AssertVerdicts(t *testing.T, got, want map[int]Verdict)

AssertVerdicts checks that received and expected verdict maps contain the same keys, and then checks that the values for these keys equal.

func BlankResultPrinter

func BlankResultPrinter(b *TestingBatch, test Test, id int)

BlankResultPrinter is the standard PrintResultFunc that outputs nothing.

func ScanConfig

func ScanConfig(text string) (m map[string]string, errs []error)

ScanConfig tries to parse a stream of key-value pairs. It expects each pair to be located on a dedicated line. Duplicate keys are allowed, the later version is preferred.

func ScanKeyValuePair

func ScanKeyValuePair(line string) (string, string, error)

ScanKeyValuePair parses the key-value pair definition of form key=value. It returns error if no equality signs are present, or if any side is empty. The space around key and value is trimmed.

func VerboseResultPrinter

func VerboseResultPrinter(b *TestingBatch, test Test, id int)

Types

type ConfigurableStopwatcher

type ConfigurableStopwatcher struct {
	// contains filtered or unexported fields
}

ConfigurableStopwatcher is an implementation of the Stopwatcher that uses real time.

func NewConfigurableStopwatcher

func NewConfigurableStopwatcher(TL time.Duration) *ConfigurableStopwatcher

NewConfigurableStopwatcher will return an initialized ConfigurableStopwatcher with the desired time limit. If time limit specified is zero or negative, the time limit will never fire.

func (*ConfigurableStopwatcher) Elapsed

func (s *ConfigurableStopwatcher) Elapsed() time.Duration

Elapsed returns the true number of seconds since the initialization of the ConfigurableStopwatcher.

func (*ConfigurableStopwatcher) TimeLimit

func (s *ConfigurableStopwatcher) TimeLimit() <-chan time.Duration

TimeLimit returns a channel that will send back the number of seconds passed since beginning until the time limit was fired. The returned value may not equal to the time limit ConfigurableStopwatcher was initialized with.

type Executable

type Executable struct {
	Path string
}

func (*Executable) Run

func (e *Executable) Run(r io.Reader, w io.Writer) error

type Inputs

type Inputs struct {
	Tests  []Test
	Config map[string]string
}

Inputs contains all information located in the inputs file. It contains all of the tests listed there and the set of key-value pairs, if were provided.

func ScanInputs

func ScanInputs(r io.Reader) (inputs Inputs, errs []error)

ScanInputs is the main routine for parsing inputs file. It splits the input by test case separator, and tries to parse each individual test case one by one. If the true first test could not be parsed without errors, it is interpreted as a configuration and parsed again. The empty tests are skipped (those that don't contain input, output and the separator). If test case could not be parsed, parsing continues to the next test case, but the errors are accumulated and returned together.

type InputsError

type InputsError string

InputsError represents an error produced while scanning inputs file.

func (InputsError) Error

func (e InputsError) Error() string

type InternalError

type InternalError bool

InternalError represents an error that occured due to internal failure in TestingBatch.

func (InternalError) Error

func (e InternalError) Error() string

type LinedError

type LinedError struct {
	Header string
	Line   int
	Err    error
}

LinedError appends line information to the error message. It is mainly used to test that errors are produced for correct lines.

func (*LinedError) Error

func (e *LinedError) Error() string

func (*LinedError) Unwrap

func (e *LinedError) Unwrap() error

type PrintResultFunc

type PrintResultFunc func(*TestingBatch, Test, int)

PrintResultFunc is type representing a function to print statistics and information about the finished test case.

type Processer

type Processer interface {
	Run(io.Reader, io.Writer) error
}

Processer interface abstracts away the concept of the executable under testing.

type ProcesserFunc

type ProcesserFunc func(io.Reader, io.Writer) error

ProcesserFunc represents an implementation of Processer that instead of a real OS-level process executes Go code.

func (ProcesserFunc) Run

func (p ProcesserFunc) Run(r io.Reader, w io.Writer) error

Run will call the underlying Go function to compute the result.

type SpyProcesser

type SpyProcesser struct {
	Proc Processer

	CallCount int
}

SpyProcesser is a test double that proxies another processer. It additionally stores the number of calls made to the Run function.

func (*SpyProcesser) Run

func (p *SpyProcesser) Run(r io.Reader, w io.Writer) error

Run will execute the Run function of the inner processer, but will also increase the call count by one.

type SpyStopwatcher

type SpyStopwatcher struct {
	TLAtCall int
	// contains filtered or unexported fields
}

SpyStopwatcher implements Stopwatcher but instead of real time substitutes index sequence numbers. If time limit equals zero, then the time limit will never fire.

func (*SpyStopwatcher) Elapsed

func (s *SpyStopwatcher) Elapsed() time.Duration

Elapsed will return the number of seconds that equals to the number of calls made to the TimeLimit method.

func (*SpyStopwatcher) TimeLimit

func (s *SpyStopwatcher) TimeLimit() <-chan time.Duration

TimeLimit returns a channel that sends the TLAtCall number of seconds back at the TLAtCall-th call to the TimeLimit method.

type Stopwatcher

type Stopwatcher interface {
	Elapsed() time.Duration
	TimeLimit() <-chan time.Duration
}

Stopwatcher abstracts away the concept of the stopwatch. At any time, one can look up the elapsed time. Additionally, one can be notified when the time is up.

type Test

type Test struct {
	Input  string
	Output string
}

Test represents a single test case: an input and the expected output.

func ScanTest

func ScanTest(str string) (Test, []error)

ScanTest parses a single test case: input and output, separated with the Input/Output separator. It also trims space around input and output. If separator is absent, it returns an error.

type TestingBatch

type TestingBatch struct {
	Stat  map[int]Verdict
	Times map[int]time.Duration

	Proc          Processer
	Swatch        Stopwatcher
	ResultPrinter PrintResultFunc
	// contains filtered or unexported fields
}

TestingBatch is responsible for running tests and evaluating the verdicts for tests. For each test case, the verdict and execution time are stored. It utilizer an instance of Processer to run tests, and an instance of Stopwatcher to track time limit. Optionally, user can set ResultPrinter to a custom function to output useful statistics about test case's result.

func NewTestingBatch

func NewTestingBatch(inputs Inputs, proc Processer, swatch Stopwatcher) *TestingBatch

NewTestingBatch will initialize channels and maps inside TestingBatch and will assign respective dependency injections.

func (*TestingBatch) Run

func (b *TestingBatch) Run()

Run will lauch test cases in parallel and then will wait for each test to finish or for the time to timelimit. When a test is finished the verdict and the time it took to execute are remembered. Additionally, ResultPrinter is called on the test case's statistics. When a time limit is reached, each not-yet-judged test is assigned TL verdict and the ResultPrinter is also called on each test.

type Verdict

type Verdict int

Verdict represents a verdict asssigned by the judge.

const (
	OK Verdict = iota
	IE
	WA
	RE
	TL
)

The set of all possible judge verdicts that can be assigned. The abbreviatons are due to competitive programming online judges. (Except for IE, that stands for Internal Error)