bench

Documentation

Overview

Package bench provides solver benchmarking utilities.

Package bench addresses the needs of solver benchmarking by:

1. providing a format for a benchmark suite. The format is command line friendly for unix commands.

2. providing a tool to create such benchmarks suites.

3. providing a format for runs on benchmarks by a solver/solver configuration.

4. providing a tool to run commands on instances in suite and output the format

5. providing a tool to query and report/compare runtimes.

Index

• func IsRunDir(root string) bool
• func IsSuiteDir(d string) bool
• func Listing(s *Suite, filt func(*Run) bool) string
• func MatchSelect(pattern string, N int, dirs ...string) []string
• func SatTotal(r *Run) int
• func Select(N int, dirs ...string) []string
• func SolvePortion(r *Run) float64
• func SolveRate(r *Run, unit time.Duration) (real float64, user float64, sys float64)
• func SolveTotal(r *Run) int
• func Summary(s *Suite) string
• func Times(r *Run) (real float64, user float64, sys float64)
• func TotalResult(r *Run, filt func(r int) bool) int
• func UnknownTotal(r *Run) int
• func UnsatTotal(r *Run) int
• type Cactus
  • func NewCactus(suite *Suite, rFilt func(*Run) bool, iFilt func(*Suite, int) bool) *Cactus
  • func (c *Cactus) Utf8(N int) string
• type InstRun
  • func NewInstRun(run *Run, inst int) (*InstRun, error)
  • func OpenInstRun(run *Run, inst int) (*InstRun, error)
• type Run
  • func NewRun(suite *Suite, name, cmd string, ito, to time.Duration) (*Run, error)
  • func OpenRun(suite *Suite, root string) (*Run, error)
  • func (r *Run) Do(i int) (*InstRun, error)
  • func (r *Run) Len() int
• type Scatter
  • func NewScatter(r1, r2 *Run, iFilt func(suite *Suite, i int) bool) *Scatter
  • func (s *Scatter) Utf8(n int) string
• type Suite
  • func CreateSuite(root string, insts []string) (*Suite, error)
  • func LinkSuite(root string, insts []string) (*Suite, error)
  • func OpenSuite(root string) (*Suite, error)
  • func (s *Suite) Len() int
  • func (s *Suite) RemoveRun(name string) error
  • func (s *Suite) Run(name, cmd string, ito, to time.Duration) (*Run, error)
  • func (s *Suite) RunSelect(filt func(*Run) bool) *Suite

Functions

func IsRunDir

func IsRunDir(root string) bool

IsRunDir tests whether or not root looks like a run directory.

func IsSuiteDir

func IsSuiteDir(d string) bool

IsSuiteDir returns true if d appears to contain a bench.Suite

func Listing

func Listing(s *Suite, filt func(*Run) bool) string

Listing produces a listing of all instances in all runs.

func MatchSelect

func MatchSelect(pattern string, N int, dirs ...string) []string

MatchSelect is like Select but filters files with filepath.Match using pattern.

func SatTotal

func SatTotal(r *Run) int

func Select

func Select(N int, dirs ...string) []string

Select walks all the directories listed in dirs and selects up to N files randomly from all files in any directory in p.

Select logs errors.

func SolvePortion

func SolvePortion(r *Run) float64

SolvePortion gives the portion of instances in r solved.

func SolveRate

func SolveRate(r *Run, unit time.Duration) (real float64, user float64, sys float64)

SolveRate gives the rate in terms of solutions found per second for the run r. It returns the results in terms of real time, user time, and system time.

SolveRate is only valid w.r.t. a instance/global timeout because unsolved instances have unknown time. SolveRate counts unsolved time.

func SolveTotal

func SolveTotal(r *Run) int

SolveTotal gives the total number of solved instances for the run r.

func Summary

func Summary(s *Suite) string

Summary produces a summary of all runs in the Suite s.

func Times

func Times(r *Run) (real float64, user float64, sys float64)

func TotalResult

func TotalResult(r *Run, filt func(r int) bool) int

func UnknownTotal

func UnknownTotal(r *Run) int

func UnsatTotal

func UnsatTotal(r *Run) int

Types

type Cactus

type Cactus struct {
	Runs   []*Run  // runs to plot.
	D      [][]int // D[i] permutation for run i instances sorted according to dur.
	MaxDur time.Duration
}

Type Cactus contains info necessary for cactus plot of an arbitrary number of runs.

func NewCactus

func NewCactus(suite *Suite, rFilt func(*Run) bool, iFilt func(*Suite, int) bool) *Cactus

NewCactus makes a new cactus object.

rFilt, if non-nil, selects the runs to show in the cactus plot

iFilt, if non-nil, selects the instances to show in the cactus plot

func (*Cactus) Utf8

func (c *Cactus) Utf8(N int) string

Utf8 produces a text image of the cactus data suitable for a utf8 monospaced font terminal.

type InstRun

type InstRun struct {
	Run    *Run
	Inst   int
	Result int
	Start  time.Time
	Dur    time.Duration
	UDur   time.Duration
	SDur   time.Duration
	Out    io.WriteCloser
	Err    io.WriteCloser
	Error  string
}

func NewInstRun

func NewInstRun(run *Run, inst int) (*InstRun, error)

func OpenInstRun

func OpenInstRun(run *Run, inst int) (*InstRun, error)

type Run

type Run struct {
	Root        string
	Name        string
	Suite       *Suite
	Cmd         string
	Desc        string
	Commit      string
	Env         map[string]string
	Arch        string
	Os          string
	NumCPU      int
	Start       time.Time
	Timeout     time.Duration
	InstTimeout time.Duration
	InstRuns    []*InstRun
}

Type Run describes a run of a command/solver on a *Suite

func NewRun

func NewRun(suite *Suite, name, cmd string, ito, to time.Duration) (*Run, error)

NewRun creates a new run for the *Suite suite rooted at root with instance timeout ito and global timeout to for command cmd.

The resulting Run is (unfortunately) not yet openable since there are no results

func OpenRun

func OpenRun(suite *Suite, root string) (*Run, error)

OpenRun opens a Run.

func (*Run) Do

func (r *Run) Do(i int) (*InstRun, error)

func (*Run) Len

func (r *Run) Len() int

type Scatter

type Scatter struct {
	Runs [2]*Run
	Xs   []time.Duration // Xs[i], Ys[i] gives point for run 0,1 on instance i.
	Ys   []time.Duration
}

Type Scatter contains info necessary for a scatter plot of 2 runs.

func NewScatter

func NewScatter(r1, r2 *Run, iFilt func(suite *Suite, i int) bool) *Scatter

NewScatter creates a new Scatter object for a pair of runs r1, r2. iFilt, if non-nil, selects the instances to show in the scatter.

func (*Scatter) Utf8

func (s *Scatter) Utf8(n int) string

Utf8 returns an ascii scatter plot with 2n columns and n rows, making up for the width/height ratio of most monospaced fonts.

type Suite

type Suite struct {
	Root   string   // root directory
	Insts  []string // instance pathnames.
	Map    []string // map from instance to origin path
	Hashes []string // map from instance to hash
	Runs   []*Run   // runs linked to this input
}

Type Suite describes the inputs to benchmark runs In general, the info in the struct is read-only.

func CreateSuite

func CreateSuite(root string, insts []string) (*Suite, error)

CreateSuite creates a benchmark suite rooted in root with the specified instances.

func LinkSuite

func LinkSuite(root string, insts []string) (*Suite, error)

LinkSuite creates a benchmark suite in which instances are symlinks to the origin files.

func OpenSuite

func OpenSuite(root string) (*Suite, error)

OpenSuite opens a benchmark suite.

func (*Suite) Len

func (s *Suite) Len() int

Len returns the number of instances in the suite.

func (*Suite) RemoveRun

func (s *Suite) RemoveRun(name string) error

RemoveRun removes a run named name from the suite s.

func (*Suite) Run

func (s *Suite) Run(name, cmd string, ito, to time.Duration) (*Run, error)

Run creates a run from a command, an instance timeout, and a global timeout.

func (*Suite) RunSelect

func (s *Suite) RunSelect(filt func(*Run) bool) *Suite

RunSelect selects a subset of runs based on a filtering function filt and returns the result in the form of a suite sr which is identical to s except that sr only contains runs in s such that filt(r) is true.