Documentation
¶
Overview ¶
Package engine orchestrates spec execution: it plans scenarios, isolates each in its own temporary workdir, materializes fixtures, runs steps in order, and aggregates results.
Index ¶
Constants ¶
View Source
const DefaultStepTimeout = "60s"
DefaultStepTimeout is the built-in bound applied to run/http/query/grpc steps when no level configures a timeout (#17), so an unconfigured hanging command fails loudly instead of stalling the run (or a CI job) forever.
Variables ¶
This section is empty.
Functions ¶
func ScenarioID ¶
ScenarioID is the stable identity of a scenario for cross-run selection: its spec path joined with its (already matrix-expanded) name. It is the key used by the --rerun-failed state file and Engine.Select (#64).
func SignalNameList ¶ added in v0.3.0
func SignalNameList() string
SignalNameList renders the accepted signal names for error messages.
Types ¶
type Counts ¶
type Counts struct {
Passed, Failed, Skipped, Errored, Flaky int
}
Counts summarizes scenario outcomes.
type Engine ¶
type Engine struct {
// OnScenario, if set, is called as soon as each scenario finishes. It lets a
// caller stream live progress (e.g. dot-style output) while a run is still
// in flight. It must not retain the value beyond the call.
OnScenario func(ScenarioResult)
// UpdateSnapshots makes snapshot assertions write the snapshot file instead
// of comparing against it.
UpdateSnapshots bool
// Parallel is the maximum number of scenarios to run concurrently. Values
// < 1 mean sequential execution.
Parallel int
// FailFast stops scheduling new scenarios once one fails or errors.
// In-flight scenarios are allowed to finish. With RetryFailed it triggers
// only on a FINAL failure (a recovered flaky scenario never trips it).
FailFast bool
// Repeat runs each selected scenario this many times (#29) to surface
// flakiness: any failing iteration fails the scenario, and the per-
// iteration statuses are recorded. Values < 2 disable it.
Repeat int
// RetryFailed re-runs a failed/errored scenario up to this many times in
// a fresh workdir (#29); a recovering re-run yields StatusFlaky — counted
// green for the exit code but reported loudly. Mutually exclusive with
// Repeat (the CLI enforces it).
RetryFailed int
// Sem, if set, is a shared concurrency limiter acquired around every
// scenario. It lets a caller run multiple suites concurrently while capping
// the TOTAL number of in-flight scenarios across all of them (a global
// worker pool). When nil, only this suite's own Parallel workers bound it.
Sem chan struct{}
// FilterName, Tags, and SkipTags select which scenarios run.
// FilterName is a substring match on the scenario name; Tags keeps only
// scenarios carrying at least one listed tag; SkipTags drops scenarios
// carrying any listed tag. Unselected scenarios are excluded entirely.
FilterName string
Tags []string
SkipTags []string
// Select, when non-nil, restricts execution to the exact scenario identities
// it contains, keyed by ScenarioID(specPath, scenarioName). It composes with
// (is intersected with) the filter/tag selection above and powers the
// `--rerun-failed` red-green loop (#64). A nil map disables identity
// selection; an empty (non-nil) map selects nothing.
Select map[string]bool
// Artifacts, when non-nil, is the directory into which failed text
// assertions write durable sidecar files for review tooling (#48, the
// --artifacts-dir flag). Nil disables artifact export.
Artifacts *artifact.Dir
// contains filtered or unexported fields
}
Engine executes specs.
type ScenarioResult ¶
type ScenarioResult struct {
Name string
// Suite is the owning suite's name, so per-scenario consumers (the
// OnScenario stream, verbose traces) can label output without threading
// the SuiteResult alongside.
Suite string
Status Status
Steps []StepResult
// Teardown records the scenario's teardown steps. They always run (pass,
// fail, error, or interrupt) and their failures are reported here, but they
// never change Status: the verdict is decided by Steps alone.
Teardown []StepResult
Duration time.Duration
SkipReason string
// SecurityViolation marks a scenario that errored because it breached the
// spec's security policy (e.g. a network-allowlist denial). It maps to exit
// code 6 rather than the generic execution-error code.
SecurityViolation bool
// ServiceLogs lists the preserved combined stdout/stderr log artifacts for
// this scenario's background services, written only when --artifacts-dir is
// set and the scenario failed or a service never became ready (#51). Paths are
// relative to the artifacts dir root.
ServiceLogs []ServiceLog
// Attempts is how many executions this result folds under --retry-failed
// (#29): 1 for a normal run, >1 when re-runs happened (StatusFlaky when
// one of them recovered). Zero means the feature was off.
Attempts int
// Iterations records each execution's status under --repeat (#29); the
// visible Steps belong to the first failing iteration (or the last one
// when all passed).
Iterations []Status
}
ScenarioResult aggregates the steps of one scenario.
func (*ScenarioResult) TeardownFailed ¶
func (s *ScenarioResult) TeardownFailed() bool
TeardownFailed reports whether any teardown step failed or errored. Reports use it to stay loud about incomplete cleanup without flipping the verdict.
type ServiceLog ¶
type ServiceLog struct {
Name string // the service's declared name
Path string // relative to the artifacts dir root, slash-separated
}
ServiceLog references one background service's preserved log artifact (#51).
type Status ¶
type Status string
Status is the outcome of a scenario or suite.
const ( // StatusPassed means every assertion passed. StatusPassed Status = "passed" // StatusFailed means at least one assertion failed. StatusFailed Status = "failed" // StatusSkipped means a skip/only condition excluded the scenario. StatusSkipped Status = "skipped" // StatusError means a step could not execute (e.g. command not found). StatusError Status = "error" // StatusFlaky means the scenario failed at least once and then passed on // a --retry-failed re-run (#29): green for the exit code, but surfaced // everywhere so instability is never silently hidden. StatusFlaky Status = "flaky" )
type StepResult ¶
type StepResult struct {
Index int
Kind spec.StepKind
Run *runner.Result // set for run steps
// Checks holds one CheckResult per assertion target evaluated for this step.
// An assert step may set several targets (exit_code + stdout + file …); each is
// an independent check. A run step's retry `until` records its checks here too.
// Empty for non-assert steps.
Checks []*assert.CheckResult
ErrMsg string // set when the step could not execute
// Setup marks an execution error that happened before any numbered step ran
// (a service-readiness failure, a workdir-creation failure). Such an error has
// no step Kind, so reports label its phase explicitly ("service setup") rather
// than emitting a blank step field or a misleading "step 0 ()".
Setup bool
}
StepResult records what happened for a single step.
type SuiteResult ¶
type SuiteResult struct {
Suite string
SpecPath string
Status Status
Scenarios []ScenarioResult
// Setup records the suite.setup steps (#7). A failed setup step errors
// every scenario (none runs) and the failure is visible here.
Setup []StepResult
// Teardown records the suite.teardown steps (#7). They always run after
// the last scenario; failures are reported but never change Status.
Teardown []StepResult
Duration time.Duration
// SecurityViolation is true when any scenario breached the security policy.
SecurityViolation bool
}
SuiteResult aggregates all scenarios of one spec file.
func (*SuiteResult) Counts ¶
func (s *SuiteResult) Counts() Counts
Counts tallies scenario statuses.
Source Files
Click to show internal directories.
Click to hide internal directories.