eval

Documentation

Overview

Package eval implements a framework for selection strategy evaluation.

For evaluation, see https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/rts/doc/eval.md

For an example using this framework, see https://source.chromium.org/chromium/infra/infra/+/HEAD:go/src/infra/rts/cmd/rts-random/main.go

Index

• func Main(ctx context.Context, strategy Strategy)
• func PrintResults(res *evalpb.Results, w io.Writer, minChangeRecall float32) error
• type Eval
  • func (e *Eval) EvaluateSafety(ctx context.Context, strategy Strategy) (*evalpb.Results, error)
  • func (e *Eval) RegisterFlags(fs *flag.FlagSet) error
  • func (e *Eval) Run(ctx context.Context, strategy Strategy) (*evalpb.Results, error)
  • func (e *Eval) ValidateFlags() error
• type Input
• type Output
• type Strategy

Functions

func Main

func Main(ctx context.Context, strategy Strategy)

Main evaluates the selection strategy, prints results and exits the process.

func PrintResults

func PrintResults(res *evalpb.Results, w io.Writer, minChangeRecall float32) error

PrintResults prints the results to w.

Types

type Eval

type Eval struct {
	// The number of goroutines to spawn for each metric.
	// If <=0, defaults to 100.
	Concurrency int

	// Rejections is a path to a directory with rejection records.
	// For format details, see comments of Rejection protobuf message.
	Rejections string

	// Durations is a path to a directory with test duration records.
	// For format details, see comments of TestDurationRecord protobuf message.
	Durations string

	// LogFurthest instructs to log rejections for which failed tests have large
	// distance, as concluded by the selection strategy.
	// LogFurthest is the number of rejections to print, ordered by descending
	// distance.
	// This can help diagnosing the selection strategy.
	//
	// TODO(nodir): implement this.
	LogFurthest int

	// LogProgressInterval indicates how often to log the number of processed
	// historical records. The field value is the number of records between
	// progress reports. If zero or less, progress is not logged.
	LogProgressInterval int
}

Eval estimates safety and efficiency of a given selection strategy.

func (*Eval) EvaluateSafety

func (e *Eval) EvaluateSafety(ctx context.Context, strategy Strategy) (*evalpb.Results, error)

EvaluateSafety evaluates the strategy's safety. The returned Result has all but efficiency-related fields populated.

func (*Eval) RegisterFlags

func (e *Eval) RegisterFlags(fs *flag.FlagSet) error

RegisterFlags registers flags for the Eval fields.

func (*Eval) Run

func (e *Eval) Run(ctx context.Context, strategy Strategy) (*evalpb.Results, error)

Run evaluates the candidate strategy.

func (*Eval) ValidateFlags

func (e *Eval) ValidateFlags() error

ValidateFlags validates values of flags registered using RegisterFlags.

type Input

type Input struct {
	// ChangedFiles is a list of changed files.
	ChangedFiles []*evalpb.SourceFile

	// The strategy needs to decide how much each of these test variants is
	// affected by the changed files.
	TestVariants []*evalpb.TestVariant
}

Input is input to a selection strategy.

type Output

type Output struct {
	// TestVariantAffectedness is how much Input.TestVariants are affected by the
	// code change, where TestVariantAffectedness[i]
	// corresponds to Input.TestVariants[i].
	//
	// When Strategy() is called, TestVariantAffectedness is pre-initialized
	// with a slice with the same length as Input.TestVariants, and zero elements.
	// Thus by default all tests are very affected (distance=0).
	TestVariantAffectedness []rts.Affectedness
}

Output is the output of a selection strategy.

type Strategy

type Strategy func(context.Context, Input, *Output) error

Strategy evaluates how much a given test is affected by given changed files.