runner

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 29, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package runner implements go test wrappers, iteration orchestration, and result analysis.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Analyze 

func Analyze(iterations []io.Reader, slowThreshold time.Duration) (*Report, LogMap, error)

Analyze reads per-iteration test2json streams and classifies tests. Malformed lines are silently skipped (go test can interleave non-JSON).

func AnalyzeResults 

func AnalyzeResults(resultsDir string, slowThreshold time.Duration) (*Report, LogMap, error)

AnalyzeResults opens every `iteration-*.log.jsonl` file in resultsDir, in numeric-iteration order, and delegates to Analyze.

func Diagnose 

func Diagnose(
	ctx context.Context,
	conf *config.App,
	out *output.Printer,
	goTestArgs []string,
	iterSetup, iterTeardown func(context.Context) error,
) error

Diagnose runs go test -json once per iteration, writing each stream to iteration-<n>.log.jsonl, then analyzes and writes report.json. With --ai-output, stdout is the results directory path during the run, per- iteration progress lines, then one JSON line with event "complete" (report path, summary, and capped findings). Full detail remains in report.json. Test iteration failures do not stop later runs (unless --fail-fast); compile/build failures stop immediately. Results are reflected in report.json. Diagnose returns a non-nil error for setup failures (e.g. mkdir, database reset), analyze/write report failures, or ctx errors bubbling from dependencies — not for failing tests alone. iterSetup and iterTeardown run before/after each iteration. Either may be nil. Teardown runs even when the iteration's go test invocation fails; its error is reported only when the iteration itself succeeded.

func EffectiveParallelIterations 

func EffectiveParallelIterations(conf *config.App) int

EffectiveParallelIterations returns the bounded diagnose worker count.

func GoTest 

func GoTest(ctx context.Context, conf *config.App, args []string) error

GoTest runs `go test` with the given args (repo root as working directory).

func Gotestsum 

func Gotestsum(ctx context.Context, conf *config.App, args []string) error

Gotestsum runs `gotestsum` with the given args (repo root as working directory).

func PrintSummary 

func PrintSummary(w io.Writer, rep *Report)

PrintSummary writes a human-readable diagnose summary: scope line and rates table first, then detail sections (Broken, Flaky, Timeout, Slow) as flat lines. Broken and Timeout entries are sorted alphabetically by package then test. Flaky entries are sorted by fails/runs (desc), then fails (desc), then name. Slow entries are sorted by max runtime (desc), then name.

func WarnDiagnoseGoTestCount 

func WarnDiagnoseGoTestCount(w io.Writer, goTestArgs []string) error

WarnDiagnoseGoTestCount prints hints when the user sets -count on go test, and returns an error if -count values in the go test flag section are malformed.

func WilsonScoreInterval 

func WilsonScoreInterval(k, n int, z float64) (lower float64, upper float64)

WilsonScoreInterval calculates our confidence interval for how (non) flaky a test is. We use a 95% confidence interval, so z = 1.96. https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

func WriteCSV 

func WriteCSV(resultsDir string, rep *Report) error

WriteCSV writes a human-readable CSV of every flagged test (Flakes ∪ Failures ∪ Timeouts ∪ Slow) to <resultsDir>/report.csv. Rows sort worst-first: (timeouts+fails) desc, then package, then test.

func WriteLogFiles 

func WriteLogFiles(resultsDir string, rep *Report, logs LogMap) error

WriteLogFiles writes per-test per-iteration log files under <resultsDir>/logs/ for flagged tests and populates each flagged TestEntry's Logs slice with a compact problem-kind, iteration-range, and path pattern.

func WriteReport 

func WriteReport(resultsDir string, rep *Report) error

WriteReport writes the report as pretty JSON to <resultsDir>/report.json.

Types

type IterationDigest 

type IterationDigest struct {
	Result       string // pass, fail, timeout
	RanTests     int    // distinct named tests (package.test) that executed (pass/fail/timeout), excluding skip-only
	FailTests    int    // len(IterationSummaries[0].FailingTests)
	TimeoutTests int    // len(Timeouts) for this iteration
	SkipTests    int    // distinct named tests skipped in this iteration
	SlowTests    int    // tests over slow threshold
	BuildFailure bool   // compile/build failed or heuristic package-level fail with no named tests run
}

IterationDigest summarizes one iteration JSONL log for per-iteration CLI output. Counts match a single-iteration Analyze (same rules as the final report).

func DigestIterationJSONL 

func DigestIterationJSONL(r io.Reader, slowThreshold time.Duration) (IterationDigest, error)

DigestIterationJSONL parses one `go test -json` stream and returns counts for progress UI. It uses the same scan + report pipeline as Analyze for one iteration (no redundant Analyze wrapper).

type IterationSummary 

type IterationSummary struct {
	Index        int           `json:"index"`
	Duration     time.Duration `json:"duration,omitempty"`
	Result       string        `json:"result"` // "pass", "fail", "timeout"
	FailingTests []string      `json:"failing_tests,omitempty"`
	ShuffleSeed  int64         `json:"shuffle_seed,omitempty"`
}

IterationSummary captures high-level stats for a single diagnose iteration. Duration and ShuffleSeed are populated by the runner after analysis.

type LogMap 

type LogMap map[testKey]map[int]string

LogMap maps (package,test) → iteration → raw interleaved output. Returned alongside Report so callers can write per-test log files without coupling the parser to the filesystem.

type ProblemLog 

type ProblemLog struct {
	Type  string `json:"type"`
	Iters string `json:"iters"`
	Path  string `json:"path"`
}

ProblemLog points to log files for iterations where this entry actually had the reported problem. Path uses "{iter}" as the iteration placeholder.

type Report 

type Report struct {
	Run                *RunMeta           `json:"run,omitempty"`
	Iterations         int                `json:"iterations"`
	SlowThreshold      time.Duration      `json:"slow_threshold"`
	Summary            *ReportSummary     `json:"summary,omitempty"`
	IterationSummaries []IterationSummary `json:"iteration_summaries,omitempty"`
	Flakes             []TestEntry        `json:"flakes,omitempty"`
	Failures           []TestEntry        `json:"failures,omitempty"`
	Timeouts           []TestEntry        `json:"timeouts,omitempty"`
	Slow               []TestEntry        `json:"slow,omitempty"`
	SlowestPackages    []TestEntry        `json:"slowest_packages,omitempty"`
}

Report classifies tests across iterations of a diagnose run.

func (*Report) TestGroups 

func (rep *Report) TestGroups() []TestGroup

TestGroups returns all flagged test categories in precedence order (Timeout > Failure > Flake > Slow) for deduplication and output generation.

type ReportSummary 

type ReportSummary struct {
	DistinctNamedTests          int      `json:"distinct_named_tests"`
	FlakeNamedCount             int      `json:"flake_named_count"`
	FlakePrevalence             *float64 `json:"flake_prevalence,omitempty"`
	FlakeFailRuns               int      `json:"flake_fail_runs,omitempty"`
	FlakeTotalRuns              int      `json:"flake_total_runs,omitempty"`
	FlakeExecutionFailRate      *float64 `json:"flake_execution_fail_rate,omitempty"`
	FlakeExecutionFailRateLower *float64 `json:"flake_execution_fail_rate_lower,omitempty"`
	FlakeExecutionFailRateUpper *float64 `json:"flake_execution_fail_rate_upper,omitempty"`
	// FlakeFailingIterations is how many diagnose iterations had at least one
	// flake failure; FlakeIterationTotal is rep.Iterations (not summed per-test runs).
	FlakeFailingIterations      int      `json:"flake_failing_iterations,omitempty"`
	FlakeIterationTotal         int      `json:"flake_iteration_total,omitempty"`
	FlakeIterationFailRate      *float64 `json:"flake_iteration_fail_rate,omitempty"`
	FlakeIterationFailRateLower *float64 `json:"flake_iteration_fail_rate_lower,omitempty"`
	FlakeIterationFailRateUpper *float64 `json:"flake_iteration_fail_rate_upper,omitempty"`
	SlowCount                   int      `json:"slow_count,omitempty"`
	SlowPrevalence              *float64 `json:"slow_prevalence,omitempty"`
	// IterationDurationMin/Max/P50 summarize wall-clock runtimes (IterationSummary.Duration) across all completed iterations.
	IterationDurationMin time.Duration `json:"iteration_duration_min,omitempty"`
	IterationDurationMax time.Duration `json:"iteration_duration_max,omitempty"`
	IterationDurationP50 time.Duration `json:"iteration_duration_p50,omitempty"`
}

ReportSummary holds aggregate flake and slow rates for the full diagnose run. FlakePrevalence uses distinct named tests (package.test keys). Per-execution flake_fail_runs / flake_total_runs sum across flaky entries; flake_failing_iterations / flake_iteration_total count diagnose iterations (union of flake failures vs rep.Iterations).

type RunMeta 

type RunMeta struct {
	ResultsDirBasename string        `json:"results_dir_basename"`
	StartedAt          time.Time     `json:"started_at"`
	FinishedAt         *time.Time    `json:"finished_at,omitempty"`
	GoTestArgs         []string      `json:"go_test_args"`
	TargetSlug         string        `json:"target_slug"`
	DiagnoseIterations int           `json:"diagnose_iterations"`
	ParallelIterations int           `json:"parallel_iterations,omitempty"`
	SlowThreshold      time.Duration `json:"slow_threshold"`
	FailFast           bool          `json:"fail_fast,omitempty"`
	FailFastOn         []string      `json:"fail_fast_on,omitempty"`
	Shuffle            bool          `json:"shuffle,omitempty"`
}

RunMeta records how the diagnose harness was invoked and where output lives. Use this for full argv and flags; the directory name only carries a short target slug and timestamp.

type TestEntry 

type TestEntry struct {
	Package       string        `json:"package"`
	Test          string        `json:"test,omitempty"`
	Runs          int           `json:"runs"`
	Successes     int           `json:"successes"`
	Fails         int           `json:"fails"`
	Skips         int           `json:"skips"`
	Timeouts      int           `json:"timeouts"`
	FailRateLower *float64      `json:"fail_rate_lower,omitempty"`
	FailRateUpper *float64      `json:"fail_rate_upper,omitempty"`
	MinElapsed    time.Duration `json:"min_elapsed"`
	MaxElapsed    time.Duration `json:"max_elapsed"`
	P50Elapsed    time.Duration `json:"p50_elapsed"`
	Logs          []ProblemLog  `json:"logs,omitempty"`
	FailIters     []int         `json:"-"`
	TimeoutIters  []int         `json:"-"`
	SlowIters     []int         `json:"-"`
}

TestEntry is a single row in the analysis report.

type TestEvent 

type TestEvent struct {
	Action      string  `json:"Action"`
	Package     string  `json:"Package"`
	Test        string  `json:"Test"`
	Elapsed     float64 `json:"Elapsed"`
	Output      string  `json:"Output"`
	FailedBuild string  `json:"FailedBuild,omitempty"`
}

TestEvent mirrors cmd/internal/test2json's TestEvent; only fields we need.

type TestGroup 

type TestGroup struct {
	Entries     *[]TestEntry
	LogKind     string
	CSVCategory string
	Iters       func(TestEntry) []int
}

TestGroup defines a category of flagged tests within a Report, exposing references for mutation and metadata for presentation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.