tefmocheck

README

Tefmocheck

Tefmocheck (Testing Failure Mode Checker) analyzes the outputs of a testing Swarming task and determines whether various failures modes occured. It produces a testing summary file (summmary.json) that contains all of the tests in the input summary, as well as a synthetic test for each failure mode starting with "testing_failure_mode/".

At most a single check (the most specific one possible) will fail on a single task. The least specific ones start with "testing_failure_mode/task_status/". This just surfaces the Swarming task status. To see further details, see the infra_and_test_std_and_klog.txt, which includes the output of the Swarming task.

This tool is invoked by the infrastructure recipes, so any changes to its interface must be soft transitions.

Test names

The tests results produced by this tool are analyzed for flakiness. Currently our flake analysis relies on seeing a test fail and then pass, which means that the test names must appear in the output summary even if they pass. This means that we cannot parse a string out of an error message and use that in a test name. https://fxbug.dev./62307 tracks improving this.

Documentation

Index

• Variables
• func RunChecks(checks []FailureModeCheck, to *TestingOutputs, outputsDir string) ([]runtests.TestDetails, error)
• type FailureModeCheck
  • func StringInLogsChecks() []FailureModeCheck
• type MassTestFailureCheck
  • func (c MassTestFailureCheck) Check(to *TestingOutputs) bool
  • func (c MassTestFailureCheck) DebugText() string
  • func (c MassTestFailureCheck) Name() string
  • func (c MassTestFailureCheck) OutputFiles() []string
• type NoTestsRanCheck
  • func (c NoTestsRanCheck) Check(to *TestingOutputs) bool
  • func (c NoTestsRanCheck) DebugText() string
  • func (c NoTestsRanCheck) Name() string
  • func (c NoTestsRanCheck) OutputFiles() []string
• type SwarmingRpcsFilesRef
• type SwarmingRpcsStringListPair
• type SwarmingRpcsTaskResult
• type SwarmingTaskSummary
  • func (sts *SwarmingTaskSummary) BotURL() string
  • func (sts *SwarmingTaskSummary) TaskURL() string
• type TestLog
  • func SplitTestLogs(logBytes []byte, logBaseName, outDir string) ([]TestLog, error)
• type TestingOutputs

Variables

var TaskStateChecks []FailureModeCheck = []FailureModeCheck{

	&taskInternalFailureCheck{},
	&taskFailureCheck{},

	&taskStateCheck{State: "BOT_DIED"},
	&taskStateCheck{State: "CANCELED"},
	&taskStateCheck{State: "EXPIRED"},
	&taskStateCheck{State: "INVALID"},
	&taskStateCheck{State: "KILLED"},
	&taskStateCheck{State: "NO_RESOURCE"},
	&taskStateCheck{State: "PENDING"},
	&taskStateCheck{State: "RUNNING"},
	&taskStateCheck{State: "TIMED_OUT"},
}

TaskStateChecks contains checks to cover every possible state. A task can only be in one state, so their relative order doesn't matter.

Functions

func RunChecks

func RunChecks(checks []FailureModeCheck, to *TestingOutputs, outputsDir string) ([]runtests.TestDetails, error)

RunChecks runs the given checks on the given TestingOutputs. A failed test means the Check() returned true. After the first failed test, all later Checks() will be skipped. Tests will not be returned for skipped or passed checks. Rationale: In order for these bugs to be useful, we want a failure to be associated only with the most specific, helpful failure modes, which then get routed to specific bugs. Hence only a single failure is returned. Rationale for not returning passed tests: We want to be able to add many checks without cluttering the output test summary with noise. Our flake detection system will identify a test that appears a a failure on one run of a swarming task, and then disappears on other runs of that same task as a flake.

Types

type FailureModeCheck

type FailureModeCheck interface {
	// Check analyzes TestingOutputs and returns true if the failure mode was detected.
	Check(*TestingOutputs) bool
	// Name is the name of this check.
	Name() string
	// DebugText is human-readable text intended to help debug a check failure.
	DebugText() string
	// OutputFiles are paths associated with this check. Often empty.
	OutputFiles() []string
}

FailureModeCheck checks whether a failure mode appears.

func StringInLogsChecks

func StringInLogsChecks() []FailureModeCheck

StringInLogsChecks returns checks to detect bad strings in certain logs.

type MassTestFailureCheck

type MassTestFailureCheck struct {
	MaxFailed int
}

func (MassTestFailureCheck) Check

func (c MassTestFailureCheck) Check(to *TestingOutputs) bool

func (MassTestFailureCheck) DebugText

func (c MassTestFailureCheck) DebugText() string

func (MassTestFailureCheck) Name

func (c MassTestFailureCheck) Name() string

func (MassTestFailureCheck) OutputFiles

func (c MassTestFailureCheck) OutputFiles() []string

type NoTestsRanCheck

type NoTestsRanCheck struct{}

NoTestsRanCheck checks whether the task reported running zero tests. It may actually have run tests but not reported them, which is still an issue.

func (NoTestsRanCheck) Check

func (c NoTestsRanCheck) Check(to *TestingOutputs) bool

func (NoTestsRanCheck) DebugText

func (c NoTestsRanCheck) DebugText() string

func (NoTestsRanCheck) Name

func (c NoTestsRanCheck) Name() string

func (NoTestsRanCheck) OutputFiles

func (c NoTestsRanCheck) OutputFiles() []string

type SwarmingRpcsFilesRef

type SwarmingRpcsFilesRef struct {
	Isolated       string `json:"isolated,omitempty"`
	Isolatedserver string `json:"isolatedserver,omitempty"`
	Namespace      string `json:"namespace,omitempty"`
}

SwarmingRpcsFilesRef is a reference to an isolated file on an isolate server.

type SwarmingRpcsStringListPair

type SwarmingRpcsStringListPair struct {
	Key   string   `json:"key,omitempty"`
	Value []string `json:"value,omitempty"`
}

SwarmingRpcsStringListPair represents a mapping of string to list of strings.

type SwarmingRpcsTaskResult

type SwarmingRpcsTaskResult struct {
	// BotDimensions: Represents a mapping of string to list of strings.
	BotDimensions    []*SwarmingRpcsStringListPair `json:"bot_dimensions,omitempty"`
	BotId            string                        `json:"bot_id,omitempty"`
	BotVersion       string                        `json:"bot_version,omitempty"`
	CurrentTaskSlice int64                         `json:"current_task_slice,omitempty,string"`
	DedupedFrom      string                        `json:"deduped_from,omitempty"`
	Duration         float64                       `json:"duration,omitempty"`
	ExitCode         int64                         `json:"exit_code,omitempty,string"`
	Failure          bool                          `json:"failure,omitempty"`
	InternalFailure  bool                          `json:"internal_failure,omitempty"`
	Name             string                        `json:"name,omitempty"`
	// OutputsRef: Defines a data tree reference for Swarming task inputs or
	// outputs. It can either be: - a reference to an isolated file on an
	// isolate server - a reference to an isolated file on a RBE CAS server
	// In the RBE CAS case, the isolatedserver must be set to GCP name, and
	// namespace must be set to "sha256-GCP". For the moment, RBE CAS
	// requires SHA-256 and doesn't support precompressed data.
	OutputsRef     *SwarmingRpcsFilesRef `json:"outputs_ref,omitempty"`
	RunId          string                `json:"run_id,omitempty"`
	ServerVersions []string              `json:"server_versions,omitempty"`
	StartedTs      string                `json:"started_ts,omitempty"`
	// Possible values:
	//   "BOT_DIED"
	//   "CANCELED"
	//   "COMPLETED"
	//   "EXPIRED"
	//   "INVALID"
	//   "KILLED"
	//   "NO_RESOURCE"
	//   "PENDING"
	//   "RUNNING"
	//   "TIMED_OUT"
	State     string   `json:"state,omitempty"`
	Tags      []string `json:"tags,omitempty"`
	TaskId    string   `json:"task_id,omitempty"`
	TryNumber int64    `json:"try_number,omitempty,string"`
	User      string   `json:"user,omitempty"`
}

SwarmingRpcsTaskResult is the result of a single task execution. This is based on the type of the same name in go.chromium.org/luci/common/api/swarming/swarming/v1, but we include only the fields we care about.

type SwarmingTaskSummary

type SwarmingTaskSummary struct {
	Outputs []string                `json:"outputs,omitempty"`
	Results *SwarmingRpcsTaskResult `json:"results"`
	Host    string
}

SwarmingTaskSummary is the summary of an individual task found in the output of `swarming collect`.

func (*SwarmingTaskSummary) BotURL

func (sts *SwarmingTaskSummary) BotURL() string

BotURL returns the Swarming UI URL for the bot the task ran on.

func (*SwarmingTaskSummary) TaskURL

func (sts *SwarmingTaskSummary) TaskURL() string

TaskURL returns the Swarming UI URL for the task.

type TestLog

type TestLog struct {
	TestName string
	Bytes    []byte
	FilePath string
}

TestLog represents an individual test's slice of a larger log file.

func SplitTestLogs

func SplitTestLogs(logBytes []byte, logBaseName, outDir string) ([]TestLog, error)

SplitTestLogs splits logBytes into per-test logs, writes those per-test logs to files in outDir, and returns a slice of TestLogs. The logs will be written to the parent directory of path.

type TestingOutputs

type TestingOutputs struct {
	TestSummary           *runtests.TestSummary
	SwarmingSummary       *SwarmingTaskSummary
	SerialLog             []byte
	SwarmingOutput        []byte
	SwarmingOutputPerTest []TestLog
	Syslog                []byte
}

TestingOutputs are the outputs of a testing Swarming task that are analyzed by a FailureModeCheck.