eval

package
Version: v0.0.0-...-e560ebb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 13, 2021 License: BSD-3-Clause Imports: 28 Imported by: 0

Documentation

Overview

Package eval implements a framework for selection strategy evaluation.

For evaluation, see https://chromium.googlesource.com/infra/luci/luci-go/+/HEAD/rts/doc/eval.md

For an example using this framework, see https://source.chromium.org/chromium/infra/infra/+/HEAD:go/src/infra/rts/cmd/rts-random/main.go

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Main

func Main(ctx context.Context, strategy Strategy)

Main evaluates the selection strategy, prints results and exits the process.

func PrintResults

func PrintResults(res *evalpb.Results, w io.Writer, minChangeRecall float32) error

PrintResults prints the results to w.

Types

type Eval

type Eval struct {
	// The number of goroutines to spawn for each metric.
	// If <=0, defaults to 100.
	Concurrency int

	// Rejections is a path to a directory with rejection records.
	// For format details, see comments of Rejection protobuf message.
	Rejections string

	// Durations is a path to a directory with test duration records.
	// For format details, see comments of TestDurationRecord protobuf message.
	Durations string

	// LogFurthest instructs to log rejections for which failed tests have large
	// distance, as concluded by the selection strategy.
	// LogFurthest is the number of rejections to print, ordered by descending
	// distance.
	// This can help diagnosing the selection strategy.
	//
	// TODO(nodir): implement this.
	LogFurthest int

	// LogProgressInterval indicates how often to log the number of processed
	// historical records. The field value is the number of records between
	// progress reports. If zero or less, progress is not logged.
	LogProgressInterval int
}

Eval estimates safety and efficiency of a given selection strategy.

func (*Eval) EvaluateSafety

func (e *Eval) EvaluateSafety(ctx context.Context, strategy Strategy) (*evalpb.Results, error)

EvaluateSafety evaluates the strategy's safety. The returned Result has all but efficiency-related fields populated.

func (*Eval) RegisterFlags

func (e *Eval) RegisterFlags(fs *flag.FlagSet) error

RegisterFlags registers flags for the Eval fields.

func (*Eval) Run

func (e *Eval) Run(ctx context.Context, strategy Strategy) (*evalpb.Results, error)

Run evaluates the candidate strategy.

func (*Eval) ValidateFlags

func (e *Eval) ValidateFlags() error

ValidateFlags validates values of flags registered using RegisterFlags.

type Input

type Input struct {
	// ChangedFiles is a list of changed files.
	ChangedFiles []*evalpb.SourceFile

	// The strategy needs to decide how much each of these test variants is
	// affected by the changed files.
	TestVariants []*evalpb.TestVariant
}

Input is input to a selection strategy.

type Output

type Output struct {
	// TestVariantAffectedness is how much Input.TestVariants are affected by the
	// code change, where TestVariantAffectedness[i]
	// corresponds to Input.TestVariants[i].
	//
	// When Strategy() is called, TestVariantAffectedness is pre-initialized
	// with a slice with the same length as Input.TestVariants, and zero elements.
	// Thus by default all tests are very affected (distance=0).
	TestVariantAffectedness []rts.Affectedness
}

Output is the output of a selection strategy.

type Strategy

type Strategy func(context.Context, Input, *Output) error

Strategy evaluates how much a given test is affected by given changed files.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL