deltacoverage

README

deltacoverage

Go package to provide delta coverage between your tests.

Installing

go install github.com/broadwing/deltacoverage/cmd/deltacoverage@latest

Example

$ deltacoverage
TestCleanup_RemovesOutputPath 0.7%
TestGenerate_CreatesExpectedCoverProfilesGivenCode 7.2%
TestGenerate_ErrorsGivenCodeWithNoTests 1.4%
TestListTests_SetsExpectedTestsNamesGivenCode 0.0%
TestNewCoverProfile_ErrorsIfPathDoesNotExist 0.7%
TestNewCoverProfile_ErrorsIfPathIsNotDirectory 0.7%
TestNewCoverProfile_SetsCorrectPackagePathGivenString 0.0%
TestNewCoverProfile_SetsNonEmptyOutputPath 0.0%
TestParse_ReturnsExpectedNumberStatementsGivenOutputPath 0.0%
TestParse_ReturnsExpectedTestsBranchesGivenOutputPath 0.0%
TestParse_ReturnsExpectedUniqueBranchesGivenOutputPath 0.0%
TestString_ReturnsExpectedDeltaCoverageGivenCoverProfile 10.1%

The test TestCleanup_RemovesOutputPath contributes to exclusive 0.7% of the total coverage while TestString_ReturnsExpectedDeltaCoverageGivenCoverProfile is responsible for 10.1%.

Although the tests TestNewCoverProfile_SetsCorrectPackagePathGivenString and TestNewCoverProfile_SetsNonEmptyOutputPath have no delta coverage or 0.0%, it is okay since we are testing behaviours and not functions. The tests run the same code path but test different things. Also, this design allow ease understand of what is the test supposed to do.

Motivation

Herb Derby came up with this metric of “delta coverage”.
You look at your test suite and you measure what coverage each test adds uniquely that no other test provides.
If you have lots of tests with no delta coverage, so that you could just delete them and not lose your
ability to exercise parts of the system, then you should delete those tests, unless they have some some
communication purpose.
—Kent Beck, “Is TDD Dead?”

Thank you @bitfield for the suggestion.

Roadmap

• POC ✅
• Brute force implementation using go test commands ✅
• Get same metrics using coverprofiles of each test instead of brute force. It's going to improve performance from O(n2) to O(n)✅
• Borrow cover.go code and create deltagecoverage.go -> Under analysis
• (Dream) Implement as Go CLI feature

Documentation

Index

• Variables
• func Main() int
• type CoverProfile
  • func NewCoverProfile(codePath string) (*CoverProfile, error)
  • func (c *CoverProfile) Cleanup() error
  • func (c *CoverProfile) Generate() error
  • func (c *CoverProfile) ListTests() error
  • func (c *CoverProfile) Parse() error
  • func (c CoverProfile) String() string

Variables

var ErrMustBeDirectory = errors.New("Must be a directory")

var ErrMustBeFile = errors.New("Must be a file")

Functions

func Main

func Main() int

Types

type CoverProfile

type CoverProfile struct {
	NumberStatements int
	OutputPath       string
	PackagePath      string
	Stderr           io.Writer
	Stdout           io.Writer
	Tests            []string
	TestsBranches    map[string][]string
	UniqueBranches   map[string]int
}

func NewCoverProfile

func NewCoverProfile(codePath string) (*CoverProfile, error)

func (*CoverProfile) Cleanup

func (c *CoverProfile) Cleanup() error

func (*CoverProfile) Generate

func (c *CoverProfile) Generate() error

func (*CoverProfile) ListTests

func (c *CoverProfile) ListTests() error

func (*CoverProfile) Parse

func (c *CoverProfile) Parse() error

Parse reads all files from a directory with extension .coverprofile, parses it, and populate CoverProfile object Design tradeoff to get the total number of statements: 1. Parse one file just to sum the total statements 2. Recalculate total statements each iteration on the loop 3. Add a counter and an if to just calculate the total sum in the first iteration Current implementation is number 3

func (CoverProfile) String

func (c CoverProfile) String() string

String prints out the deltacoverage percentage for each test