e2e

README

pkg/e2e

e2e runs a declarative end-to-end test against a real Kubernetes cluster. Give it a spec file and it orchestrates the full lifecycle — cluster creation, CRD apply, operator install, CR apply, expectation checking, and cleanup — the same way locally and in CI.

ork e2e -f e2e.yaml
ork e2e ./...                    # discover and run all *e2e.yaml files recursively
ork e2e ./examples/beginner/...  # scoped discovery
ork e2e init                     # scaffold e2e.yaml from the current Katalog
ork e2e init --suite             # write a suite aggregator from discovered leaf files

Developer documentation

I want to…	Go to
Understand the spec file format and all fields	docs/01-spec.md
Understand how expectations are evaluated (resources, commands, polling)	docs/02-expectations.md
Understand the full run pipeline (what happens in order)	docs/03-pipeline.md
Understand cluster lifecycle (kind, reuse, context restore, shared Orkestra)	docs/04-cluster.md
Compose test suites with imports and the wait: field	docs/05-imports.md
Use ./... discovery mode, --wait, --skip, --dry-run	docs/06-discovery.md
Test any Kubernetes workload without Orkestra (custom.target: kubernetes)	docs/07-kubernetes-target.md

---

Adding a new kubectl: subcommand

[!IMPORTANT] Every new kubectl: subcommand requires changes in five places and a fixture entry. Missing any one of them means the feature is incomplete.

1. Type — pkg/types/e2e.go

Add a struct for the new subcommand. Assertion fields are the same on every subcommand — copy them verbatim:

type E2EKubectlMyCmd struct {
    // required fields...
    Equals            string `yaml:"equals,omitempty"`
    NotEquals         string `yaml:"notEquals,omitempty"`
    OutputContains    string `yaml:"outputContains,omitempty"`
    OutputNotContains string `yaml:"outputNotContains,omitempty"`
    GreaterThan       string `yaml:"greaterThan,omitempty"`
    LessThan          string `yaml:"lessThan,omitempty"`
}

Add a slice for it on E2EKubectl:

MyCmd []E2EKubectlMyCmd `yaml:"my-cmd,omitempty"`

2. Runner — pkg/e2e/verify.go

Add checkKubectlMyCmd(ctx, e, workDir) and call it from checkKubectl. Use the assertions struct when calling applyAssertions — do not pass positional strings:

for i, e := range k.MyCmd {
    if err := checkKubectlMyCmd(ctx, e, workDir); err != nil {
        return fmt.Errorf("kubectl.my-cmd[%d]: %w", i, err)
    }
}

// inside checkKubectlMyCmd:
if err := applyAssertions(out, assertions{
    Equals:            e.Equals,
    NotEquals:         e.NotEquals,
    OutputContains:    e.OutputContains,
    OutputNotContains: e.OutputNotContains,
    GreaterThan:       e.GreaterThan,
    LessThan:          e.LessThan,
}); err != nil {
    return err
}

Mutations (apply, patch) run before reads (get, logs, describe, exec, port-forward) so changes take effect before assertions check them.

3. Validator — pkg/e2e/validate.go

Add validateKubectlMyCmd(loc, e) and call it from ValidateKubectl:

for j, e := range exp.Kubectl.MyCmd {
    errs = append(errs, validateKubectlMyCmd(loc("my-cmd", j), e)...)
}

4. Schema doc — documentation/reference/schema/04-e2e/07-kubectl.md

Add a ## kubectl.my-cmd section with the field table and a usage example.

5. hasKubectl count — cmd/cli/validate.go

Add len(k.MyCmd) to the kubectlCount sum in cmd/cli/validate.go:

var kubectlCount int
if k := exp.Kubectl; k != nil {
    kubectlCount = len(k.Get) + len(k.Logs) + ... + len(k.MyCmd)
}

Without this, an expectation that uses only the new subcommand fails validation with "must have at least one resource, command, or kubectl check" even though the YAML is correct.

6. Fixture — pkg/e2e/fixture/e2e.yaml

Add a checkpoint that exercises the new subcommand end-to-end against the E2EProbe operator. See fixture/README.md for details.

Documentation

Overview

Package e2e implements the orchestration loop for `ork e2e`.

A Runner executes a declarative E2E spec through its full lifecycle:

1. Cluster provisioning (kind) — skipped when --use-current or --cluster is set
2. CRD apply
3. Optional setup manifests
4. Bundle generate + apply
5. Orkestra helm install
6. CR apply
7. Expectation polling
8. Teardown — always runs for non-owned clusters (--use-current, --cluster); for owned clusters only when --keep-cluster is absent

Teardown reverses every applied resource in the correct order: CR delete → helm uninstall → bundle delete → setup helm (reverse) → setup files (reverse) → CRDs. This keeps borrowed clusters clean regardless of pass/fail.

Run returns a *Result with per-case timings that callers (e.g. registry push) embed as OCI annotations.

Index

• Constants
• func BuildDiscoveryE2E(paths []string, wait string) orktypes.E2E
• func DiscoverE2EFiles(root string, skip []string) ([]string, error)
• func DiscoverSimulateFiles(root string, skip []string) ([]string, error)
• func ExpandExpectIncludes(expects []orktypes.E2EExpectation, baseDir string) ([]orktypes.E2EExpectation, error)
• func ValidateImports(baseDir string, imports []orktypes.E2EImport) []error
• func ValidateKubectl(expects []orktypes.E2EExpectation) []error
• type CaseResult
• type ImportResult
  • func (i ImportResult) Name() string
  • func (i ImportResult) Passed() bool
• type Result
  • func (r *Result) AllPassed() bool
  • func (r *Result) Duration() string
  • func (r *Result) Passed() int
  • func (r *Result) Summary() string
  • func (r *Result) Total() int
• type Runner
  • func New(e2eFile, clusterCtx string, useCurrentCtx, keepCluster, devServer bool, ...) (*Runner, error)
  • func (r *Runner) Run(ctx context.Context) (*Result, error)

Constants

const (

	// DevServerAddr is the in-cluster DNS address CR specs should use.
	DevServerAddr = "http://orkestra-dev-server.orkestra-system.svc:9999"
)

Functions

func BuildDiscoveryE2E

func BuildDiscoveryE2E(paths []string, wait string) orktypes.E2E

BuildDiscoveryE2E constructs an in-memory E2E aggregator from a list of discovered file paths. wait is injected on every import except the first.

func DiscoverE2EFiles

func DiscoverE2EFiles(root string, skip []string) ([]string, error)

DiscoverE2EFiles walks root recursively and returns paths to all *e2e.yaml files that are not pure aggregators (files with imports: but no spec:). Results are sorted for deterministic order. skip is a list of glob patterns (matched against the full path); any file whose path matches is excluded.

func DiscoverSimulateFiles

func DiscoverSimulateFiles(root string, skip []string) ([]string, error)

DiscoverSimulateFiles walks root recursively and returns paths to all simulate.yaml leaf files (files with a spec:, not pure aggregators). Results are sorted for deterministic order.

func ExpandExpectIncludes

func ExpandExpectIncludes(expects []orktypes.E2EExpectation, baseDir string) ([]orktypes.E2EExpectation, error)

ExpandExpectIncludes resolves include: entries in the expect list, loading each referenced file and splicing its entries in place. Relative paths are resolved against baseDir. Nested includes are not supported.

func ValidateImports

func ValidateImports(baseDir string, imports []orktypes.E2EImport) []error

ValidateImports checks that every file listed in imports exists and declares kind: E2E. baseDir is the directory that relative paths are resolved against. Returns one error per invalid import — callers may print all of them.

func ValidateKubectl

func ValidateKubectl(expects []orktypes.E2EExpectation) []error

ValidateKubectl checks every kubectl block across all expect entries. Returns one error per violation — callers collect and print all of them.

Types

type CaseResult

type CaseResult struct {
	Name    string
	Passed  bool
	Elapsed time.Duration
	Err     error
}

CaseResult holds the outcome of a single expectation.

type ImportResult

type ImportResult struct {
	// Path is the import path as declared in imports: (e.g. "./01-basic/e2e.yaml").
	Path string
	// Result is the structured outcome. Nil when the import failed to load or
	// the runner itself errored before producing any expectations.
	Result *Result
	// Err is non-nil when the import failed (load error, Orkestra not ready,
	// or one or more expectations failed).
	Err error
}

ImportResult holds the outcome of one imported E2E file in a suite run.

func (ImportResult) Name

func (i ImportResult) Name() string

Name returns the display name — the E2E metadata.name when available, otherwise the import path.

func (ImportResult) Passed

func (i ImportResult) Passed() bool

Passed reports whether this import succeeded.

type Result

type Result struct {
	Name    string
	Cases   []CaseResult
	Elapsed time.Duration
}

Result holds the outcome of a complete E2E run. It is returned by Run and consumed by ork push to embed verification metadata into OCI annotations.

func (*Result) AllPassed

func (r *Result) AllPassed() bool

AllPassed returns true when every expectation passed.

func (*Result) Duration

func (r *Result) Duration() string

Duration returns the total elapsed time as a human-readable string (e.g. "45s").

func (*Result) Passed

func (r *Result) Passed() int

Passed returns the number of passing expectations.

func (*Result) Summary

func (r *Result) Summary() string

Summary returns a one-line result string, e.g. "3 of 3 passed (12s)".

func (*Result) Total

func (r *Result) Total() int

Total returns the total number of expectations.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes a single E2E spec end-to-end.

func New

func New(e2eFile, clusterCtx string, useCurrentCtx, keepCluster, devServer bool, orkestraVersion string, valueFiles []string, helmArgs ...string) (*Runner, error)

New loads an E2E spec from a YAML file and constructs a Runner.

func (*Runner) Run

func (r *Runner) Run(ctx context.Context) (*Result, error)

Run executes the full E2E test pipeline and returns a structured Result.