testrunner

package
v0.1.0-alpha.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 30, 2025 License: Apache-2.0 Imports: 25 Imported by: 0

README

testrunner - Validation Test Runner

Pure component for executing embedded validation tests defined in HAProxyTemplateConfig CRDs.

Overview

The test runner executes validation tests that verify template rendering and HAProxy configuration correctness. Tests are embedded directly in HAProxyTemplateConfig CRDs and can be run via:

  • CLI: controller validate command for local development and CI/CD
  • Webhook: Automatic validation during admission control

The test runner is a pure component with no EventBus dependency, designed for direct function calls from both CLI and webhook contexts.

Usage

Basic Test Execution
import (
    "context"
    "haptic/pkg/controller/testrunner"
    "haptic/pkg/templating"
    "haptic/pkg/dataplane"
)

// 1. Create template engine
templates := map[string]string{
    "haproxy.cfg": config.HAProxyConfig.Template,
}
engine, err := templating.New(templating.EngineTypeScriggo, templates, nil, nil, nil)
if err != nil {
    return err
}

// 2. Configure validation paths
validationPaths := dataplane.ValidationPaths{
    HAProxyBinary:    "/usr/sbin/haproxy",
    TempDir:          "/tmp/haproxy-validation",
    AuxiliaryFileDir: "/tmp/haproxy-validation/aux",
}

// 3. Create test runner
runner := testrunner.New(
    config,                  // *v1alpha1.HAProxyTemplateConfigSpec
    engine,                  // *templating.TemplateEngine
    validationPaths,         // dataplane.ValidationPaths
    testrunner.Options{
        Logger: logger,      // *slog.Logger (optional)
    },
)

// 4. Run all tests
ctx := context.Background()
results, err := runner.RunTests(ctx, "")
if err != nil {
    return fmt.Errorf("test execution failed: %w", err)
}

// 5. Check results
if !results.AllPassed() {
    fmt.Printf("%d/%d tests failed\n", results.FailedTests, results.TotalTests)
    for _, test := range results.TestResults {
        if !test.Passed {
            fmt.Printf("  FAIL: %s - %s\n", test.TestName, test.RenderError)
            for _, assertion := range test.Assertions {
                if !assertion.Passed {
                    fmt.Printf("    ✗ %s: %s\n", assertion.Type, assertion.Error)
                }
            }
        }
    }
}
Running Specific Test
// Run only the test named "basic-rendering"
results, err := runner.RunTests(ctx, "basic-rendering")
if err != nil {
    return fmt.Errorf("test 'basic-rendering' not found: %w", err)
}

API Reference

Types
Runner
type Runner struct {
    // Contains private fields
}

Main test runner that executes validation tests.

Constructor:

func New(
    config *v1alpha1.HAProxyTemplateConfigSpec,
    engine *templating.TemplateEngine,
    validationPaths dataplane.ValidationPaths,
    options Options,
) *Runner
Options
type Options struct {
    // Logger for structured logging. If nil, uses default logger.
    Logger *slog.Logger
}

Configuration options for the test runner.

OutputOptions
type OutputOptions struct {
    // Format specifies output format (summary, json, yaml)
    Format OutputFormat

    // Verbose enables showing rendered content previews for failed assertions
    Verbose bool
}

Controls output formatting and verbosity for test results.

Example:

// Standard output
output, _ := testrunner.FormatResults(results, testrunner.OutputOptions{
    Format: testrunner.OutputFormatSummary,
})

// Verbose output with content previews
output, _ := testrunner.FormatResults(results, testrunner.OutputOptions{
    Format:  testrunner.OutputFormatSummary,
    Verbose: true,
})
TestResults
type TestResults struct {
    // TotalTests is the total number of tests executed.
    TotalTests int

    // PassedTests is the number of tests that passed all assertions.
    PassedTests int

    // FailedTests is the number of tests with at least one failed assertion.
    FailedTests int

    // TestResults contains detailed results for each test.
    TestResults []TestResult

    // Duration is the total time taken to run all tests.
    Duration time.Duration
}

// AllPassed returns true if all tests passed.
func (r *TestResults) AllPassed() bool

Aggregated results from running validation tests.

TestResult
type TestResult struct {
    // TestName is the name of the test.
    TestName string

    // Description is the test description.
    Description string

    // Passed is true if all assertions passed.
    Passed bool

    // Duration is the time taken to run this test.
    Duration time.Duration

    // Assertions contains results for each assertion.
    Assertions []AssertionResult

    // RenderError is set if template rendering failed.
    RenderError string

    // Rendered content (populated for observability)
    RenderedConfig string              // HAProxy configuration
    RenderedMaps   map[string]string   // Map files (path → content)
    RenderedFiles  map[string]string   // General files (filename → content)
    RenderedCerts  map[string]string   // SSL certificates (path → content)
}

Result of running a single validation test.

Rendered Content Fields: Populated after successful rendering for debugging with --dump-rendered flag or programmatic access. Empty if rendering fails.

AssertionResult
type AssertionResult struct {
    // Type is the assertion type (haproxy_valid, contains, etc).
    Type string

    // Description is the assertion description.
    Description string

    // Passed is true if the assertion passed.
    Passed bool

    // Error contains the failure message if assertion failed.
    Error string

    // Target metadata (populated for observability)
    Target        string  // Target name (e.g., "map:path-prefix.map")
    TargetSize    int     // Content size in bytes
    TargetPreview string  // First 200 chars (failed assertions only)
}

Result of running a single assertion.

Target Metadata Fields: Populated automatically for all assertions to aid debugging. Content preview only populated for failed assertions when verbose mode is enabled.

Methods
RunTests
func (r *Runner) RunTests(ctx context.Context, testName string) (*TestResults, error)

Executes validation tests.

Parameters:

  • ctx - Context for cancellation and timeouts
  • testName - Name of specific test to run (empty string runs all tests)

Returns:

  • *TestResults - Aggregated test results
  • error - Fatal error (not test failures)

Behavior:

  1. Filters tests if testName is specified
  2. For each test:
    • Creates resource stores from fixtures
    • Renders HAProxy configuration with fixtures
    • Runs all assertions against rendered output
  3. Returns aggregated results

Example:

// Run all tests
results, err := runner.RunTests(ctx, "")

// Run specific test
results, err := runner.RunTests(ctx, "my-test")

Test Execution Flow

1. Filter Tests
   └─ Select tests to run (all or specific by name)

2. For Each Test:
   ├─ Create Fixture Stores
   │  ├─ Parse fixtures from CRD
   │  ├─ Create indexed stores
   │  └─ Populate with test data
   │
   ├─ Render Configuration
   │  ├─ Build template context with fixture stores
   │  ├─ Render haproxy.cfg
   │  └─ Render auxiliary files (maps, files, certs)
   │
   ├─ Run Assertions
   │  ├─ haproxy_valid: Validate config syntax
   │  ├─ contains: Pattern matching
   │  ├─ not_contains: Pattern absence
   │  ├─ equals: Exact value comparison
   │  └─ jsonpath: JSONPath queries
   │
   └─ Collect Results
      ├─ Test passed/failed
      ├─ Assertion details
      └─ Error messages

3. Aggregate Results
   ├─ Total/passed/failed counts
   ├─ Individual test results
   └─ Total duration

Assertion Types

haproxy_valid

Validates HAProxy configuration syntax using the HAProxy binary.

Example:

assertions:
  - type: haproxy_valid
    description: "Config must be syntactically valid"

Validation Process:

  1. Write config to temporary file
  2. Write auxiliary files to temp directory
  3. Execute: haproxy -c -f <config-file>
  4. Parse HAProxy output for errors
  5. Return simplified error message if invalid
contains

Checks if target content contains a regex pattern.

Example:

assertions:
  - type: contains
    target: haproxy.cfg
    pattern: "backend api-.*"
    description: "Config should contain API backends"

Target Options:

  • haproxy.cfg - Main HAProxy configuration (default)
  • map:<name> - Map file content
  • file:<name> - General file content
  • cert:<name> - SSL certificate content
not_contains

Ensures target content does NOT contain a pattern.

Example:

assertions:
  - type: not_contains
    target: haproxy.cfg
    pattern: "debug mode"
    description: "Production config should not have debug mode"
equals

Verifies target content exactly equals expected value.

Example:

assertions:
  - type: equals
    target: map:backends.map
    expected: |
      api 10.0.0.1:8080
      web 10.0.0.2:80
    description: "Backend map should have correct entries"
jsonpath

Queries template context using JSONPath expressions.

Example:

assertions:
  - type: jsonpath
    jsonpath: "{.resources.services[0].metadata.name}"
    expected: "api-service"
    description: "First service should be api-service"

JSONPath Syntax:

  • .resources.services - Access service store
  • .resources.ingresses[0] - First ingress
  • .metadata.name - Resource field

Fixtures

Fixtures are test data injected into resource stores for template rendering.

Fixture Format
validationTests:
  - name: "with-services"
    fixtures:
      services:
        - metadata:
            name: api
            namespace: default
          spec:
            clusterIP: 10.0.0.1
            ports:
              - port: 80
        - metadata:
            name: web
            namespace: default
          spec:
            clusterIP: 10.0.0.2
    assertions:
      - type: contains
        pattern: "backend default-api"
Fixture Processing
  1. Indexing: Fixtures are indexed using the same IndexBy expressions as production watchers
  2. TypeMeta: APIVersion and Kind are inferred if not provided
  3. Store Creation: Each resource type gets its own indexed store
  4. Template Access: Accessible via resources.<type>.List() in templates
Template Usage
{%- for _, svc := range resources.services.List() %}
backend {{ svc.metadata.namespace }}-{{ svc.metadata.name }}
  server {{ svc.metadata.name }} {{ svc.spec.clusterIP }}:80
{%- end %}

Error Messages

Rendering Errors

Template rendering errors are simplified for user-friendliness:

Raw Error:

failed to render template 'haproxy.cfg': unable to execute template:
failed to call function 'fail': Service 'api' not found in namespace 'default'

Simplified:

Service 'api' not found in namespace 'default'
Validation Errors

HAProxy validation errors are simplified:

Raw Error:

[ALERT] 350/123456 (12345) : parsing [/tmp/haproxy.cfg:15] :
'maxconn' : integer expected, got 'invalid' (line 15, column 12)

Simplified:

maxconn: integer expected, got 'invalid' (line 15)

Output Formats

Summary Format (Default)

Human-readable output with pass/fail symbols:

Test Results
============

✓ basic-rendering (12ms)
  ✓ haproxy_valid: Config is syntactically valid
  ✓ contains: Backend section present

✗ with-invalid-service (8ms)
  ✗ haproxy_valid: Service 'nonexistent' not found

Total: 2 tests, 1 passed, 1 failed (20ms)
JSON Format

Structured output for CI/CD integration:

{
  "totalTests": 2,
  "passedTests": 1,
  "failedTests": 1,
  "duration": "20ms",
  "testResults": [
    {
      "testName": "basic-rendering",
      "passed": true,
      "duration": "12ms",
      "assertions": [...]
    }
  ]
}
YAML Format

Structured output with better readability:

totalTests: 2
passedTests: 1
failedTests: 1
duration: 20ms
testResults:
  - testName: basic-rendering
    passed: true
    duration: 12ms
    assertions: [...]

Observability

The test runner provides rich observability features to help debug failing tests.

Enhanced Error Messages

All assertions include helpful context in error messages by default:

✗ Path map must use MULTIBACKEND qualifier
  Error: pattern "..." not found in map:path-prefix.map (target size: 61 bytes).
         Hint: Use --verbose to see content preview
Verbose Mode

Enable verbose mode to see content previews for failed assertions:

output, err := testrunner.FormatResults(results, testrunner.OutputOptions{
    Format:  testrunner.OutputFormatSummary,
    Verbose: true,
})

Output includes:

  • Target name and size
  • First 200 characters of content
  • Hint about --dump-rendered for full content
Accessing Rendered Content

Programmatically access rendered content for debugging:

results, err := runner.RunTests(ctx, "")

for _, test := range results.TestResults {
    if !test.Passed {
        // Access rendered HAProxy config
        fmt.Printf("Config:\n%s\n", test.RenderedConfig)

        // Access map files
        for mapName, content := range test.RenderedMaps {
            fmt.Printf("Map %s:\n%s\n", mapName, content)
        }

        // Access general files
        for filename, content := range test.RenderedFiles {
            fmt.Printf("File %s:\n%s\n", filename, content)
        }

        // Access certificates
        for certPath, content := range test.RenderedCerts {
            fmt.Printf("Cert %s:\n%s\n", certPath, content)
        }
    }
}
Accessing Assertion Metadata

Access detailed metadata for failed assertions:

for _, assertion := range test.Assertions {
    if !assertion.Passed {
        fmt.Printf("Assertion: %s\n", assertion.Description)
        fmt.Printf("Type: %s\n", assertion.Type)
        fmt.Printf("Target: %s (%d bytes)\n", assertion.Target, assertion.TargetSize)

        if assertion.TargetPreview != "" {
            fmt.Printf("Preview:\n%s\n", assertion.TargetPreview)
        }
    }
}
Template Tracing

Enable template tracing for execution visibility:

// Enable tracing on engine
engine.EnableTracing()

// Run tests
runner := testrunner.New(config, engine, paths, options)
results, _ := runner.RunTests(ctx, "")

// Get trace output
trace := engine.GetTraceOutput()
fmt.Println(trace)

Trace output:

Rendering: haproxy.cfg
Completed: haproxy.cfg (0.007ms)
Rendering: path-prefix.map
Completed: path-prefix.map (3.347ms)

Performance

Typical Performance
  • Single test: <10ms (without HAProxy validation)
  • HAProxy validation: +50-200ms per test
  • Memory usage: 1-5MB per test (depends on fixture size)
Optimization Tips
  1. Minimize Fixtures: Only include resources needed for the test
  2. Skip HAProxy Validation: Use contains assertions for faster feedback during development
  3. Parallel Execution: Tests are independent and can run concurrently (future enhancement)

Limitations

Current Limitations
  1. No Template Caching: Each test renders templates independently
  2. Sequential Execution: Tests run sequentially (parallel execution planned)
  3. No Fixture Sharing: Fixtures are recreated for each test
  4. HAProxy Binary Required: haproxy_valid assertions require HAProxy binary
Future Enhancements
  • Parallel test execution
  • Template compilation caching
  • Fixture store reuse
  • Additional assertion types (regex_capture, json_schema)
  • Test dependencies (run tests in order)

See Also

  • User Documentation: docs/validation-tests.md - Guide for writing validation tests
  • Development Context: CLAUDE.md - Detailed development information
  • DryRunValidator: pkg/controller/dryrunvalidator/ - Webhook integration
  • CLI Command: cmd/controller/validate.go - CLI integration

Documentation

Overview

Package testrunner implements validation test execution for HAProxyTemplateConfig.

This package provides a test runner that executes embedded validation tests defined in HAProxyTemplateConfig CRDs. It can be used both by the CLI (controller validate command) and by the admission webhook for validation.

The test runner:

  • Creates resource stores from test fixtures
  • Renders templates with fixture context
  • Runs assertions against rendered output
  • Returns structured test results

This is a pure component with no EventBus dependency - it's called directly by the CLI and by the DryRunValidator component.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateHTTPStoreFromFixtures 

func CreateHTTPStoreFromFixtures(fixtures []config.HTTPResourceFixture, logger *slog.Logger) *httpstore.HTTPStore

CreateHTTPStoreFromFixtures creates an HTTPStore pre-populated with fixture content.

Parameters:

  • fixtures: HTTP fixtures from test definition
  • logger: Logger for debug messages

Returns:

  • HTTPStore with fixtures loaded as accepted content

func FormatResults 

func FormatResults(results *TestResults, options OutputOptions) (string, error)

FormatResults formats test results according to the specified options.

func MergeFixtures 

func MergeFixtures(globalFixtures, testFixtures map[string][]interface{}) map[string][]interface{}

MergeFixtures deep merges global fixtures with test fixtures by resource identity.

Resource identity is defined by: apiVersion + kind + namespace + name

Merge strategy:

  • Global fixtures are added first
  • Test fixtures override global fixtures when same resource identity exists
  • Resource types not in global fixtures are taken from test fixtures

Parameters:

  • globalFixtures: Fixtures from validationTests._global
  • testFixtures: Fixtures from specific test

Returns:

  • Merged fixtures map (resource type → list of resources)

func MergeHTTPFixtures 

func MergeHTTPFixtures(globalFixtures, testFixtures []config.HTTPResourceFixture) []config.HTTPResourceFixture

MergeHTTPFixtures merges global and test-specific HTTP fixtures.

Test-specific fixtures override global fixtures for the same URL.

Parameters:

  • globalFixtures: HTTP fixtures from _global test
  • testFixtures: HTTP fixtures from specific test

Returns:

  • Merged HTTP fixtures list

Types

type AssertionResult 

type AssertionResult struct {
	// Type is the assertion type (haproxy_valid, contains, etc).
	Type string

	// Description is the assertion description.
	Description string

	// Passed is true if the assertion passed.
	Passed bool

	// Error contains the failure message if assertion failed.
	Error string

	// Target is the assertion target (e.g., "haproxy.cfg", "map:path-prefix.map").
	Target string `json:"target,omitempty" yaml:"target,omitempty"`

	// TargetSize is the size of the target content in bytes.
	TargetSize int `json:"targetSize,omitempty" yaml:"targetSize,omitempty"`

	// TargetPreview is a preview of the target content (first 200 chars, only for failed assertions).
	TargetPreview string `json:"targetPreview,omitempty" yaml:"targetPreview,omitempty"`
}

AssertionResult contains the result of running a single assertion.

type FixtureHTTPStoreWrapper 

type FixtureHTTPStoreWrapper struct {
	// contains filtered or unexported fields
}

FixtureHTTPStoreWrapper wraps an HTTPStore pre-populated with fixtures for test execution.

Unlike the production HTTPStoreWrapper, this wrapper:

  • Returns content only for URLs that have fixtures loaded
  • Returns an error for URLs without fixtures (ensuring all HTTP dependencies are mocked)
  • Does NOT make actual HTTP requests

Template usage: 

{{ http.Fetch("http://blocklist.example.com/list.txt") }}

If the URL is not in fixtures, the template fails with a clear error message.

func NewFixtureHTTPStoreWrapper 

func NewFixtureHTTPStoreWrapper(store *httpstore.HTTPStore, logger *slog.Logger) *FixtureHTTPStoreWrapper

NewFixtureHTTPStoreWrapper creates a new fixture-only HTTP wrapper.

Parameters:

  • store: HTTPStore pre-populated with fixtures via LoadFixture()
  • logger: Logger for debug messages

func (*FixtureHTTPStoreWrapper) Fetch 

func (w *FixtureHTTPStoreWrapper) Fetch(args ...interface{}) (interface{}, error)

Fetch returns fixture content for a URL.

Template usage (same as production wrapper): 

{{ http.Fetch("http://example.com/data.txt") }}
{{ http.Fetch("http://example.com/data.txt", {"delay": "5m"}) }}

In fixture mode:

  • Options (delay, timeout, etc.) are ignored
  • Authentication is ignored
  • Only the URL is used to look up fixture content
  • Returns error if URL is not in fixtures

Returns:

  • Content string if URL has fixture
  • Error if URL is not in fixtures

type Options 

type Options struct {
	// TestName filters tests to run. If empty, all tests run.
	TestName string

	// Logger for structured logging. If nil, uses default logger.
	Logger *slog.Logger

	// Workers is the number of parallel workers for test execution.
	// Default: 4
	// Set to 1 for sequential execution.
	Workers int

	// DebugFilters enables detailed filter operation logging.
	// When enabled, each sort comparison is logged with values and results.
	DebugFilters bool

	// ProfileIncludes enables include timing profiling.
	// When enabled, shows which included templates take the most time.
	ProfileIncludes bool

	// Capabilities defines which features are available for the local HAProxy version.
	// Used to determine path resolution (e.g., CRT-list paths fallback when not supported).
	Capabilities dataplane.Capabilities
}

Options configures the test runner.

type OutputFormat 

type OutputFormat string

OutputFormat specifies the output format for test results. 

const (
	// OutputFormatSummary outputs a human-readable summary.
	OutputFormatSummary OutputFormat = "summary"

	// OutputFormatJSON outputs structured JSON.
	OutputFormatJSON OutputFormat = "json"

	// OutputFormatYAML outputs structured YAML.
	OutputFormatYAML OutputFormat = "yaml"
)

type OutputOptions 

type OutputOptions struct {
	// Format specifies the output format (summary, json, yaml).
	Format OutputFormat

	// Verbose enables showing rendered content previews for failed assertions.
	Verbose bool
}

OutputOptions configures output formatting.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes validation tests for HAProxyTemplateConfig.

It's a pure component with no EventBus dependency, designed to be called directly from the CLI or from the DryRunValidator.

func New 

func New(
	cfg *config.Config,
	engine templating.Engine,
	validationPaths *dataplane.ValidationPaths,
	options Options,
) *Runner

New creates a new test runner.

Parameters:

  • cfg: The internal config containing templates and validation tests
  • engine: Pre-compiled template engine
  • validationPaths: Filesystem paths for HAProxy validation
  • options: Runner options

Returns:

  • A new Runner instance ready to execute tests

func (*Runner) CreateStoresFromFixtures 

func (r *Runner) CreateStoresFromFixtures(fixtures map[string][]interface{}) (map[string]types.Store, error)

CreateStoresFromFixtures creates resource stores from test fixtures.

This converts the test fixtures (map of resource type → list of resources) into resource stores that can be used for template rendering.

Implementation:

  • Phase 1: Create empty stores for ALL watched resources
  • Phase 2: Populate stores with fixture data where provided

This ensures templates can safely call .List() on any watched resource type, even if that resource type is not present in test fixtures.

Parameters:

  • fixtures: Map of resource type names to lists of Kubernetes resources

Returns:

  • Map of resource type names to resource stores
  • error if fixture processing fails

func (*Runner) RunTests 

func (r *Runner) RunTests(ctx context.Context, testName string) (*TestResults, error)

RunTests executes all validation tests (or a specific test if filtered).

This method:

  1. Filters tests if a specific test name was requested
  2. For each test: - Creates resource stores from fixtures - Renders HAProxy configuration - Runs all assertions
  3. Aggregates and returns results

Parameters:

  • ctx: Context for cancellation and timeouts

Returns:

  • TestResults containing results for all executed tests
  • error if a fatal error occurred (not test failures)

type TestResult 

type TestResult struct {
	// TestName is the name of the test.
	TestName string

	// Description is the test description.
	Description string

	// Passed is true if all assertions passed.
	Passed bool

	// Duration is the time taken to run this test.
	Duration time.Duration

	// Assertions contains results for each assertion.
	Assertions []AssertionResult

	// RenderError is set if template rendering failed.
	RenderError string

	// RenderedConfig contains the rendered HAProxy configuration (for --dump-rendered).
	RenderedConfig string `json:"renderedConfig,omitempty" yaml:"renderedConfig,omitempty"`

	// RenderedMaps contains rendered map files (for --dump-rendered).
	RenderedMaps map[string]string `json:"renderedMaps,omitempty" yaml:"renderedMaps,omitempty"`

	// RenderedFiles contains rendered general files (for --dump-rendered).
	RenderedFiles map[string]string `json:"renderedFiles,omitempty" yaml:"renderedFiles,omitempty"`

	// RenderedCerts contains rendered SSL certificates (for --dump-rendered).
	RenderedCerts map[string]string `json:"renderedCerts,omitempty" yaml:"renderedCerts,omitempty"`

	// IncludeStats contains timing statistics for included templates (for --profile-includes).
	IncludeStats []templating.IncludeStats `json:"includeStats,omitempty" yaml:"includeStats,omitempty"`
}

TestResult contains the result of running a single validation test.

type TestResults 

type TestResults struct {
	// TotalTests is the total number of tests executed.
	TotalTests int

	// PassedTests is the number of tests that passed all assertions.
	PassedTests int

	// FailedTests is the number of tests with at least one failed assertion.
	FailedTests int

	// TestResults contains detailed results for each test.
	TestResults []TestResult

	// Duration is the total time taken to run all tests.
	Duration time.Duration
}

TestResults contains the results of running validation tests.

func (*TestResults) AllPassed 

func (r *TestResults) AllPassed() bool

AllPassed returns true if all tests passed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.