runner

Documentation

Overview

Package runner provides the TestRunner orchestration engine and RetryController for resilient test execution across all dimensions.

The runner package is Tier 4 in the dependency order (spec-v2 §5). It imports:

• Tier 0: types, result
• Tier 1: transport, config, registry
• Tier 3: dimensions (DimensionRunner interface)
• Tier 2: store (baseline updates after performance tests)

Index

• type RetryConfig
  • func DefaultRetryConfig() RetryConfig
• type RetryController
  • func NewRetryController(cfg RetryConfig) *RetryController
  • func (r *RetryController) Run(ctx context.Context, tc *registry.TestCase, ...) (*result.TestResult, error)
• type RunnerConfig
• type TestRunner
  • func NewRunner(cfg RunnerConfig, runners map[types.Dimension]dimensions.DimensionRunner, ...) *TestRunner
  • func (r *TestRunner) Run(ctx context.Context, suite *registry.TestSuite) (*result.TestRun, error)

Types

type RetryConfig

type RetryConfig struct {
	MaxAttempts  int           `yaml:"maxAttempts"  json:"maxAttempts"`
	BaseDelay    time.Duration `yaml:"baseDelay"     json:"baseDelay"`
	MaxDelay     time.Duration `yaml:"maxDelay"      json:"maxDelay"`
	JitterFactor float64       `yaml:"jitterFactor"  json:"jitterFactor"`
}

RetryConfig controls the retry behavior of RetryController.

func DefaultRetryConfig

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns retry configuration with sensible defaults: 1 attempt (no retries), 100ms base delay, 5s max delay, 20% jitter. Per D-04, the caller overrides MaxAttempts based on tc.Retries+1.

type RetryController

type RetryController struct {
	// contains filtered or unexported fields
}

RetryController wraps test case execution with exponential backoff retry logic. It retries only on retriable failures (StatusErrored and transport transient errors), never on permanent failures (StatusFailed, StatusPassed).

func NewRetryController

func NewRetryController(cfg RetryConfig) *RetryController

NewRetryController creates a RetryController with the given configuration.

func (*RetryController) Run

func (r *RetryController) Run(ctx context.Context, tc *registry.TestCase, fn func(ctx context.Context, tc *registry.TestCase) (*result.TestResult, error)) (*result.TestResult, error)

Run executes fn with retry logic. It calls fn(ctx, tc) up to cfg.MaxAttempts times. Retries only occur for retriable conditions (see isRetriable). On context cancellation during backoff, it returns immediately with ctx.Err(). On exhausted attempts, it returns the last result/error.

type RunnerConfig

type RunnerConfig struct {
	NumWorkers  int               `yaml:"numWorkers"    json:"numWorkers"`
	Dimensions  []types.Dimension `yaml:"dimensions"    json:"dimensions"`
	Tags        []string          `yaml:"tags"          json:"tags"`
	Verbose     bool              `yaml:"verbose"       json:"verbose"`
	RetryConfig RetryConfig       `yaml:"retryConfig"   json:"retryConfig"`
}

RunnerConfig holds configuration for TestRunner.

type TestRunner

type TestRunner struct {
	// contains filtered or unexported fields
}

TestRunner orchestrates test execution across dimensions using a worker pool for non-performance tests and sequential execution for performance tests.

Per spec §19 and §23:

• Non-perf dimensions (functional, routing, clarity) execute in a worker pool of K = min(numCPU*2, 20) goroutines.
• Non-perf workers drain naturally via channel close before performance tests start.
• Performance tests execute sequentially after the non-perf pool completes.
• Dimension runners (routing, clarity) acquire the shared LLM semaphore internally; the worker does NOT acquire it. This avoids deadlock from nested semaphore acquisition.

func NewRunner

func NewRunner(cfg RunnerConfig, runners map[types.Dimension]dimensions.DimensionRunner, st store.Store) *TestRunner

NewRunner creates a TestRunner with the given configuration, dimension runners, and store.

func (*TestRunner) Run

func (r *TestRunner) Run(ctx context.Context, suite *registry.TestSuite) (*result.TestRun, error)

Run executes all test cases in the suite according to the concurrency model:

1. Non-perf test cases (functional, routing, clarity) dispatch to a worker pool.
2. Worker pool drains naturally via channel close.
3. Performance test cases execute sequentially.
4. Summary is computed with ScoreAggregator.

Per D-15, a dimension with zero test cases passes its gate (vacuous pass).