lexecutor

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

README

lexecutor

lexecutor is a Go test harness that runs AMPEL policy test suites against multiple ampel versions at once: the current development tree (via the ampel Go library) plus the two most recent stable ampel releases (via downloaded binaries). It lets a policy repository verify that its policies keep producing the expected PASS/FAIL verdicts across the versions of ampel that consumers are actually running.

It is the engine behind the test suite in the community policies repository, but it works with any directory tree of policies and test definitions.

How it works

Point lexecutor at a directory and it will:

  1. Discover every .ptests.yaml suite under that directory.
  2. Resolve runners — one per ampel version:
    • HEAD: the ampel version compiled into your test binary, exercised through the Go library (policyctl's tester).
    • stable / eol: the two most recent stable ampel release tags. Their binaries are downloaded from GitHub releases (falling back to building from source), then invoked as subprocesses. This needs the gh CLI to be available and authenticated.
  3. Run every test case in every suite against every runner, as parallel Go subtests, and assert each verdict matches the declared expectation.

If the versioned binaries can't be resolved (for example gh is unavailable), the harness logs a warning and runs against HEAD only.

Usage

Add a single Go test that hands your policy tree to lexecutor:

package policies_test

import (
	"testing"

	"github.com/carabiner-dev/lexecutor"
)

func TestPolicies(t *testing.T) {
	lexecutor.RunAllTests(t, ".")
}

go test ./...

To run against HEAD only (or a custom set of versions), call RunAllTestsWithRunners with your own []VersionRunner instead.

Test suites: .ptests.yaml

Each .ptests.yaml file declares a list of test cases. Paths are resolved relative to the file's directory.

tests:
  - name: no-critical-passes-when-only-medium-present
    policy: osv-no-critical.hjson          # policy or policy set to evaluate
    expect: PASS                           # PASS or FAIL
    subject: "sha256:7950b24d0640..."      # subject digest (algo:hex)
    attestations:
      - ../testdata/osv/no-critical.intoto.json
    context:                               # optional context values
      - name: max_severity
        value: high
    context-files:                         # optional context providers
      - path: ../testdata/context.json
    ampel-version: v1.3.7                  # optional: minimum ampel version

Runtime requirements and version skipping

A policy can declare that it needs a specific evaluator engine version or CEL plugin through its runtime specifier, e.g.:

meta: {
  runtime: "cel@v1?plugin:osv=v1"
}

Older ampel releases that predate a plugin can't evaluate such a policy. For each test, lexecutor reads the plugin requirements declared by the policy and skips the test on any runner that can't provide them, rather than reporting a spurious failure:

  • HEAD (the development tree) is assumed to carry every plugin the repository's policies target, so it always runs them.
  • A released binary is considered capable only if its ampel verify --help advertises --skip-unsupported-runtime. Capable binaries are asked to soft-fail (skip) unmet policies via that flag; a SOFTFAIL verdict is reported as a skip. Binaries that predate the flag have the test skipped before it runs.

Policies with no plugin requirements run on every runner, unchanged.

Version floors: ampel-version

Some behavior changes with the ampel release rather than with a plugin, for example a bug fix in a bundled library that flips a policy's verdict. A test case can declare the minimum ampel version it needs:

  - name: producers-passes-with-cyclonedx-suppliers
    policy: protobom-sbom-producers.hjson
    ampel-version: v1.3.7
    expect: PASS

lexecutor compares that floor against each runner's engine version and skips the test on older engines instead of failing it:

  • stable / eol report the release tag their binary was built from.
  • HEAD reports the ampel module version linked into the test binary, read from its build info. Pseudo-versions compare as semver, so a build of a commit between two tags sorts between them. When the version is unknown (for example a replace directive pointing at a local checkout reports (devel)), HEAD runs every test.

Since eol is always the second most recent release, a floored test starts running on both released binaries as soon as the next ampel version ships.

License

Apache-2.0. See LICENSE.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RunAllTests 

func RunAllTests(t *testing.T, rootDir string)

RunAllTests discovers test suites under rootDir and runs each test case as a Go subtest against HEAD (library), stable, and eol (downloaded or built from the two most recent upstream tags).

func RunAllTestsWithRunners 

func RunAllTestsWithRunners(t *testing.T, rootDir string, runners []VersionRunner)

RunAllTestsWithRunners discovers suites and runs them against the provided version runners.

Types

type AmpelBinaries 

type AmpelBinaries struct {
	// Stable is the binary for the latest stable tag.
	Stable string
	// StableTag is the tag used for the stable binary.
	StableTag string
	// EOL is the binary for the second-latest stable tag.
	EOL string
	// EOLTag is the tag used for the eol binary.
	EOLTag string
	// contains filtered or unexported fields
}

AmpelBinaries holds the paths to ampel binaries for specific versions.

func GetAmpelBinaries 

func GetAmpelBinaries() (*AmpelBinaries, error)

GetAmpelBinaries fetches ampel binaries for the two most recent stable tags. It downloads pre-built binaries from GitHub releases. If that fails, it falls back to cloning and building from source.

func (*AmpelBinaries) Cleanup 

func (b *AmpelBinaries) Cleanup()

Cleanup removes the temporary directory with the binaries.

type BinaryRunner 

type BinaryRunner struct {
	// Name is the version label shown in test output (e.g. "stable", "eol").
	Name string

	// Tag is the ampel release tag the binary was built from (e.g. "v1.3.7").
	// It is the engine version compared against a test case's ampel-version
	// floor. Leave empty when unknown; the runner then runs every test.
	Tag string

	// BinaryPath is the path to the ampel binary to invoke.
	BinaryPath string
	// contains filtered or unexported fields
}

BinaryRunner shells out to an ampel binary for verification. This avoids Go module dependency conflicts when testing against older ampel versions.

func (*BinaryRunner) EngineVersion 

func (b *BinaryRunner) EngineVersion() string

EngineVersion returns the release tag the binary was built from.

func (*BinaryRunner) RunTest 

func (b *BinaryRunner) RunTest(ctx context.Context, baseDir string, tc *tester.TestCase) (*tester.TestResult, error)

func (*BinaryRunner) SupportsCollectors 

func (b *BinaryRunner) SupportsCollectors(context.Context) bool

SupportsCollectors reports whether this ampel binary can run collector-based tests. Such tests need the binary's collector to synthesize the evidence (e.g. build a signature attestation from a sigstore bundle or a detached certificate + signature pair), and a binary whose collector predates that support silently returns no attestations, which surfaces as a spurious failure rather than a skip.

Unlike plugin requirements, that capability has no CLI signal to probe (the --collector and --signer flags long predate it), so released binaries are gated out and collector-based tests run against HEAD only. Once binaries advertise the capability, probe for it here instead of returning false.

func (*BinaryRunner) SupportsRuntimeRequirements 

func (b *BinaryRunner) SupportsRuntimeRequirements(ctx context.Context) bool

SupportsRuntimeRequirements probes (once) whether the binary advertises the --skip-unsupported-runtime verify flag. Presence of that flag is our signal that this ampel version understands policy runtime/plugin requirements and will skip rather than choke on a policy whose plugins it lacks. Older binaries that predate the flag return false.

func (*BinaryRunner) Version 

func (b *BinaryRunner) Version() string

type DiscoveredSuite 

type DiscoveredSuite struct {
	Dir   string            // directory containing the config file
	Suite *tester.TestSuite // parsed config
}

DiscoveredSuite is a test suite found during discovery.

func Discover 

func Discover(rootDir string) ([]DiscoveredSuite, error)

Discover walks rootDir looking for .ptests.yaml files and returns the parsed suites.

type HeadRunner 

type HeadRunner struct {
	// contains filtered or unexported fields
}

HeadRunner uses the current (HEAD) ampel version via the Go library.

func (*HeadRunner) EngineVersion 

func (h *HeadRunner) EngineVersion() string

EngineVersion returns the version of the ampel module linked into the test binary as recorded in its build info: a release tag such as "v1.3.6", or a pseudo-version for an untagged commit. It returns "" when there is no build info or the module isn't linked, and "(devel)" when a replace directive points at a local checkout. TestCase.RunsOn treats both as unknown, so version-floored tests still run against them.

func (*HeadRunner) RunTest 

func (h *HeadRunner) RunTest(ctx context.Context, baseDir string, tc *tester.TestCase) (*tester.TestResult, error)

func (*HeadRunner) SupportsCollectors 

func (h *HeadRunner) SupportsCollectors(context.Context) bool

SupportsCollectors is always true for HEAD: the library under test is the current development tree, which carries the collector features the tests in this repository target.

func (*HeadRunner) SupportsRuntimeRequirements 

func (h *HeadRunner) SupportsRuntimeRequirements(context.Context) bool

SupportsRuntimeRequirements is always true for HEAD: the library under test is the current development tree, which carries every plugin the policies in this repository target.

func (*HeadRunner) Version 

func (h *HeadRunner) Version() string

type VersionRunner 

type VersionRunner interface {
	Version() string
	RunTest(ctx context.Context, baseDir string, tc *tester.TestCase) (*tester.TestResult, error)

	// EngineVersion returns the semantic version of the ampel engine this
	// runner executes (e.g. "v1.3.7"), or "" when it can't be determined.
	// It is compared against a test case's ampel-version floor: tests that
	// need a newer engine are skipped on this runner. An unknown version
	// runs every test.
	EngineVersion() string

	// SupportsRuntimeRequirements reports whether this ampel understands policy
	// runtime/plugin requirements, i.e. it will run a policy that declares them
	// (when it has the plugin) or gracefully skip it (when it doesn't) rather
	// than fail with an undefined-function error. Tests whose policy declares
	// plugin requirements are skipped on runners that return false.
	SupportsRuntimeRequirements(ctx context.Context) bool

	// SupportsCollectors reports whether this runner can execute test cases that
	// declare `collectors:` — i.e. whether it can synthesize evidence (such as
	// signature attestations built from sigstore bundles or detached
	// signatures) by running a collector. Tests that declare collectors are
	// skipped on runners that return false.
	SupportsCollectors(ctx context.Context) bool
}

VersionRunner can execute a test case against a specific ampel version.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.