testing

package
v0.0.235 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 22, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

## Service Testing - **Lifecycle Management**: Test service creation, management, and deletion - **Dependency Management**: Validate service dependency resolution - **State Transitions**: Test service state changes and event handling - **Integration**: Test service interaction with other system components

Mock MCP Server Support

The testing framework includes comprehensive mock MCP server functionality:

- **Mock Tool Definitions**: Define tools with configurable responses - **Conditional Responses**: Different responses based on input args - **Template-Based Responses**: Dynamic response generation using Go templates - **Error Simulation**: Simulate various error conditions and failures - **Standalone Mode**: Run mock servers independently for external testing

Mock servers can be embedded in test scenarios or run standalone: 

# Embedded in test scenario
mcpServers:
  - name: "mock-kubernetes"
    type: "mock"
    tools:
      - name: "kubectl_get_pods"
        responses:
          - condition: {namespace: "default"}
            response: "pod1\npod2\npod3"

# Standalone mode
muster test --mock-mcp-server --mock-config=mock-config.yaml

Extension Points

The testing framework is designed for extensibility through several mechanisms:

## Custom Test Steps - Implement custom step types by extending the TestStep interface - Register custom step handlers with the TestRunner - Support for domain-specific validation logic

## Custom Reporters - Implement the TestReporter interface for custom output formats - Support for custom metrics collection and analysis - Integration with external monitoring and alerting systems

## Custom Scenario Loaders - Implement the TestScenarioLoader interface for alternative configuration sources - Support for dynamic scenario generation - Integration with external test management systems

Thread Safety

The testing framework is designed with thread safety as a core requirement:

- **Concurrent Execution**: All components support concurrent access - **Immutable Data**: Test scenarios and configuration are immutable after loading - **Thread-Safe Reporting**: Reporter implementations handle concurrent access - **Resource Isolation**: Each test execution maintains isolated state

**Safety Guarantees**: - TestRunner can safely execute multiple scenarios concurrently - MCPTestClient instances are thread-safe for concurrent tool calls - TestReporter can safely handle concurrent progress and result reporting - Shared resources (connections, files) are protected with appropriate synchronization

Performance Characteristics

The framework is optimized for both development and CI/CD environments:

## Execution Performance - **Parallel Execution**: Configurable worker pools (1-10 workers recommended) - **Connection Pooling**: Reuse MCP connections across multiple test steps - **Lazy Loading**: Scenarios loaded only when needed - **Memory Efficiency**: Streaming result processing for large test suites

## Scaling Characteristics - **Linear Scaling**: Performance scales linearly with parallel worker count - **Memory Usage**: O(n) memory usage relative to active scenarios - **Network Efficiency**: Connection reuse and request batching - **Resource Cleanup**: Automatic cleanup prevents resource leaks

**Performance Recommendations**: - Use 2-4 parallel workers for development testing - Use 4-8 parallel workers for CI/CD environments - Implement timeouts appropriate for your environment (30s-5m recommended) - Monitor memory usage with large test suites (>100 scenarios)

Usage Patterns

## Basic Testing 

```bash
muster test                          # Run all tests
muster test --category=behavioral    # Category-specific
muster test --concept=serviceclass  # Concept-specific
muster test --scenario=basic-create # Scenario-specific
```

## Advanced Configuration 

```bash
muster test --parallel=50 --timeout=10m --fail-fast
muster test --verbose --debug --output-format=json
```

## CI/CD Integration 

```bash
muster test --category=integration --output-format=junit --report-file=results.xml
```

## Mock MCP Server Testing 

```bash
# Run with mock servers defined in scenarios
muster test --scenario=test-with-mock

# Run standalone mock server
muster test --mock-mcp-server --mock-config=mock-tools.yaml
```

Dependencies

The testing framework requires: - A running muster aggregator server (muster serve) - MCP protocol communication capabilities - Access to all core_* MCP tools exposed by the aggregator - Write access for temporary files and test artifacts

Error Handling

The framework implements comprehensive error handling: - **Graceful Degradation**: Continue execution when possible - **Detailed Error Messages**: Context-rich error information - **Error Classification**: Distinguish between test failures and framework errors - **Recovery Mechanisms**: Automatic retry and fallback logic where appropriate

Integration with CI/CD

The framework provides first-class CI/CD support: - **Structured Output**: JSON, JUnit XML, and TAP output formats - **Exit Codes**: Proper exit codes for automation (0=success, 1=failures, 2=errors) - **Report Files**: File-based output for artifact collection - **Environment Integration**: Support for CI environment variables and configuration

For complete usage documentation, see docs/testing/README.md For scenario authoring guidance, see docs/testing/scenarios.md For comprehensive examples, see docs/testing/examples/

Package testing provides test helper tools for BDD test scenarios. These tools allow test scenarios to interact with mock infrastructure components that run in the test orchestrator process.

Index

Constants

View Source 
const (
	// TestToolSimulateOAuthCallback simulates completing an OAuth flow for testing.
	TestToolSimulateOAuthCallback = "test_simulate_oauth_callback"
	// TestToolInjectToken directly injects an access token for testing.
	TestToolInjectToken = "test_inject_token"
	// TestToolGetOAuthServerInfo returns information about mock OAuth servers.
	TestToolGetOAuthServerInfo = "test_get_oauth_server_info"
	// TestToolAdvanceOAuthClock advances the mock OAuth server's clock for testing.
	TestToolAdvanceOAuthClock = "test_advance_oauth_clock"
	// TestToolReadAuthStatus reads the auth://status resource to verify auth state.
	TestToolReadAuthStatus = "test_read_auth_status"
	// TestToolRevokeToken revokes a token on the mock OAuth server for testing.
	TestToolRevokeToken = "test_revoke_token"
	// TestToolCreateUser creates a new user session for multi-user testing.
	TestToolCreateUser = "test_create_user"
	// TestToolSwitchUser switches to a different user session for multi-user testing.
	TestToolSwitchUser = "test_switch_user"
	// TestToolListToolsForUser lists tools visible to a specific user.
	TestToolListToolsForUser = "test_list_tools_for_user"
	// TestToolGetCurrentUser returns the current active user name.
	TestToolGetCurrentUser = "test_get_current_user"
	// TestToolSimulateMusterReauth simulates re-authentication to muster with a new token
	// while preserving the session ID. Used to test proactive SSO re-triggering.
	TestToolSimulateMusterReauth = "test_simulate_muster_reauth"
	// TestToolMusterAuthLogin simulates `muster auth login` by completing the OAuth
	// callback to muster's token store. This is required for SSO token forwarding tests.
	TestToolMusterAuthLogin = "test_muster_auth_login"
)

Test tool name constants for BDD test scenarios.

Variables

This section is empty.

Functions

func CleanupStaleMusterTestProcesses 

func CleanupStaleMusterTestProcesses(logger TestLogger, debug bool)

CleanupStaleMusterTestProcesses kills any muster processes that were left behind from previous test runs. It identifies test processes by looking for muster serve commands with --config-path pointing to /tmp/muster-test-* directories.

This should be called at the start of each test suite to ensure a clean slate. The function is best-effort and logs errors rather than returning them, since cleanup failures should not block test execution.

func FormatValidationResults 

func FormatValidationResults(results *ScenarioValidationResults, verbose bool) string

FormatValidationResults formats validation results for CLI output

func GenerateScenarioPrefix 

func GenerateScenarioPrefix(scenarioName string) string

GenerateScenarioPrefix generates a short unique prefix for a scenario. The prefix is formatted as [XXXXXX] where XXXXXX is derived from the scenario name. For readability, it combines: - First 3 chars of the cleaned scenario name - First 3 chars of a SHA256 hash of the full name (for uniqueness)

func GetDefaultScenarioPath 

func GetDefaultScenarioPath() string

GetDefaultScenarioPath returns the default path for test scenarios

func GetScenarioPath 

func GetScenarioPath(configPath string) string

GetScenarioPath determines the actual scenario path to use, handling empty/default cases

func IsTestTool 

func IsTestTool(toolName string) bool

IsTestTool returns true if the tool name is a test helper tool.

func LoadSchemaFromFile 

func LoadSchemaFromFile(filename string) (map[string]interface{}, error)

LoadSchemaFromFile loads a JSON schema from file

Types

type ExecutionMode 

type ExecutionMode string

ExecutionMode represents the mode of test execution 

const (
	// ExecutionModeCLI represents command line interface execution
	ExecutionModeCLI ExecutionMode = "cli"
	// ExecutionModeMCPServer represents MCP server execution via stdio
	ExecutionModeMCPServer ExecutionMode = "mcp-server"
)

type InstanceLogs 

type InstanceLogs struct {
	// Stdout contains the standard output
	Stdout string
	// Stderr contains the standard error output
	Stderr string
	// Combined contains both stdout and stderr in chronological order
	Combined string
}

InstanceLogs contains the captured logs from an muster instance

type MCPServerConfig 

type MCPServerConfig struct {
	// Name is the unique identifier for the MCP server
	Name string `yaml:"name"`
	// Config contains the server-specific configuration (can include tools for mock servers)
	Config map[string]interface{} `yaml:"config"`
}

MCPServerConfig represents an MCP server configuration

type MCPServerOAuthConfig 

type MCPServerOAuthConfig struct {
	// Required indicates this server requires OAuth authentication
	Required bool `yaml:"required"`

	// MockOAuthServerRef references a mock OAuth server by name
	MockOAuthServerRef string `yaml:"mock_oauth_server_ref"`

	// Scope is the required OAuth scope
	Scope string `yaml:"scope,omitempty"`
}

MCPServerOAuthConfig defines OAuth protection for an MCP server in tests

type MCPTestClient 

type MCPTestClient interface {
	// Connect establishes connection to the MCP aggregator
	Connect(ctx context.Context, endpoint string) error
	// ConnectWithAuth establishes connection to the MCP aggregator with an access token.
	// This is used when muster's OAuth server is enabled and requires authentication.
	ConnectWithAuth(ctx context.Context, endpoint, accessToken string) error
	// CallTool invokes an MCP tool with the given args
	CallTool(ctx context.Context, toolName string, args map[string]interface{}) (interface{}, error)
	// ListTools returns available MCP tools
	ListTools(ctx context.Context) ([]string, error)
	// ListToolsWithSchemas returns available MCP tools with their full schemas
	ListToolsWithSchemas(ctx context.Context) ([]mcp.Tool, error)
	// ReadResource reads an MCP resource by URI
	ReadResource(ctx context.Context, uri string) (*mcp.ReadResourceResult, error)
	// Close closes the MCP connection
	Close() error
	// GetSessionID returns the client's session ID (may be empty if not set)
	GetSessionID() string
	// ReconnectWithSession reconnects using a specific session ID and new access token.
	// This is used to test proactive SSO re-triggering when a token changes for the same session.
	ReconnectWithSession(ctx context.Context, endpoint, accessToken, sessionID string) error
}

MCPTestClient interface defines the MCP client for testing

func NewMCPTestClient 

func NewMCPTestClient(debug bool) MCPTestClient

NewMCPTestClient creates a new MCP test client

func NewMCPTestClientWithLogger 

func NewMCPTestClientWithLogger(debug bool, logger TestLogger) MCPTestClient

NewMCPTestClientWithLogger creates a new MCP test client with custom logger

type MainConfig 

type MainConfig struct {
	// Config contains the main configuration
	Config map[string]interface{} `yaml:"config"`
}

MainConfig represents the main muster configuration

type MockHTTPServerInfo 

type MockHTTPServerInfo struct {
	// Name is the name of the MCP server
	Name string
	// Port is the port the mock HTTP server is listening on
	Port int
	// Transport is the transport type (sse or streamable-http)
	Transport string
	// Endpoint is the full URL endpoint for the server
	Endpoint string
}

MockHTTPServerInfo contains information about a running mock HTTP server

type MockOAuthServerConfig 

type MockOAuthServerConfig struct {
	// Name is the unique identifier for this OAuth server
	Name string `yaml:"name"`

	// Issuer is the OAuth issuer URL (auto-generated if not specified)
	Issuer string `yaml:"issuer,omitempty"`

	// Scopes are the scopes this OAuth server accepts
	Scopes []string `yaml:"scopes,omitempty"`

	// AutoApprove automatically approves authentication requests
	AutoApprove bool `yaml:"auto_approve,omitempty"`

	// PKCERequired enforces PKCE flow
	PKCERequired bool `yaml:"pkce_required,omitempty"`

	// TokenLifetime is how long tokens are valid (e.g., "1h", "30m")
	TokenLifetime string `yaml:"token_lifetime,omitempty"`

	// ClientID is the expected OAuth client ID
	ClientID string `yaml:"client_id,omitempty"`

	// ClientSecret is the expected OAuth client secret
	ClientSecret string `yaml:"client_secret,omitempty"`

	// SimulateError can simulate error conditions
	SimulateError string `yaml:"simulate_error,omitempty"`

	// UseMockClock enables a mock clock for testing token expiry
	// When true, the test_advance_oauth_clock tool can be used to
	// advance time without waiting
	UseMockClock bool `yaml:"use_mock_clock,omitempty"`

	// UseAsMusterOAuthServer configures this mock OAuth server as muster's
	// OAuth server (aggregator.oauthServer), enabling SSO token forwarding
	// tests where muster itself requires OAuth authentication.
	// When true, the aggregator will be configured with oauthServer.enabled=true
	// and the issuer will be set to this mock server's URL.
	UseAsMusterOAuthServer bool `yaml:"use_as_muster_oauth_server,omitempty"`

	// TrustedIssuers configures RFC 8693 token exchange support.
	// When configured, this OAuth server can exchange tokens from the listed
	// issuers for tokens valid on this server. This enables cross-cluster SSO testing.
	// Each entry maps a connector_id to a trusted OAuth server name.
	TrustedIssuers []TrustedIssuerConfig `yaml:"trusted_issuers,omitempty"`

	// UseTLS enables HTTPS mode with a self-signed certificate.
	// This is required for OAuth servers used as token exchange targets,
	// since RFC 8693 token exchange endpoints must use HTTPS for security.
	// Note: This is automatically set to true when UseAsMusterOAuthServer is true.
	UseTLS bool `yaml:"use_tls,omitempty"`
}

MockOAuthServerConfig defines a mock OAuth server for testing

type MockOAuthServerInfo 

type MockOAuthServerInfo struct {
	// Name is the unique identifier for this OAuth server
	Name string
	// Port is the port the OAuth server is listening on
	Port int
	// IssuerURL is the OAuth issuer URL
	IssuerURL string
	// AccessToken is a pre-generated access token for test framework authentication.
	// This is only set when the OAuth server is used as muster's OAuth server
	// (UseAsMusterOAuthServer=true), allowing the test framework to authenticate
	// with muster without implementing the full OAuth browser flow.
	AccessToken string
	// IDToken is the ID token from the OAuth response.
	// This is used for SSO token forwarding tests.
	IDToken string
	// UseAsMusterOAuthServer indicates this OAuth server is used as muster's OAuth server.
	UseAsMusterOAuthServer bool
}

MockOAuthServerInfo contains info about a running mock OAuth server

type MusterInstance 

type MusterInstance struct {
	// ID is the unique identifier for this instance
	ID string
	// ConfigPath is the path to the temporary configuration directory
	ConfigPath string
	// Port is the port the instance is listening on
	Port int
	// Endpoint is the full MCP endpoint URL
	Endpoint string
	// Process is the running muster serve process
	Process *os.Process
	// StartTime when the instance was started
	StartTime time.Time
	// Logs contains the collected stdout and stderr from the instance
	Logs *InstanceLogs
	// ExpectedTools contains the list of tools expected to be available from MCP servers
	ExpectedTools []string
	// ExpectedServiceClasses contains the list of ServiceClasses expected to be available
	ExpectedServiceClasses []string
	// ExpectedMCPServers contains the list of MCP server names expected to be registered.
	// This includes OAuth-protected servers which may be in "auth_required" state.
	// Used by WaitForReady to ensure servers are registered before tests run.
	ExpectedMCPServers []string
	// MockHTTPServers contains references to mock HTTP servers started for this instance
	MockHTTPServers map[string]*MockHTTPServerInfo
	// MockOAuthServers contains references to mock OAuth servers started for this instance
	MockOAuthServers map[string]*MockOAuthServerInfo
	// MusterOAuthAccessToken is the access token for authenticating with muster's OAuth server.
	// This is set when a mock OAuth server is configured with UseAsMusterOAuthServer=true.
	// The test framework uses this token to authenticate with muster without a browser flow.
	MusterOAuthAccessToken string
}

MusterInstance represents a managed muster serve instance

type MusterInstanceManager 

type MusterInstanceManager interface {
	// CreateInstance creates a new muster serve instance with the given configuration.
	// The logger parameter allows scenario-specific logging with prefixes for parallel execution.
	CreateInstance(ctx context.Context, scenarioName string, config *MusterPreConfiguration, logger TestLogger) (*MusterInstance, error)
	// DestroyInstance stops and cleans up an muster serve instance.
	// The logger parameter allows scenario-specific logging with prefixes for parallel execution.
	DestroyInstance(ctx context.Context, instance *MusterInstance, logger TestLogger) error
	// WaitForReady waits for an instance to be ready to accept connections.
	// The logger parameter allows scenario-specific logging with prefixes for parallel execution.
	WaitForReady(ctx context.Context, instance *MusterInstance, logger TestLogger) error
}

MusterInstanceManager manages muster serve instances for testing

func NewMusterInstanceManagerWithConfig 

func NewMusterInstanceManagerWithConfig(debug bool, basePort int, logger TestLogger, keepTempConfig bool) (MusterInstanceManager, error)

NewMusterInstanceManagerWithConfig creates a new muster instance manager with custom logger and config options

func NewMusterInstanceManagerWithLogger 

func NewMusterInstanceManagerWithLogger(debug bool, basePort int, logger TestLogger) (MusterInstanceManager, error)

NewMusterInstanceManagerWithLogger creates a new muster instance manager with custom logger

type MusterPreConfiguration 

type MusterPreConfiguration struct {
	// MCPServers defines MCP server configurations to load
	MCPServers []MCPServerConfig `yaml:"mcp_servers,omitempty"`
	// Workflows defines workflow definitions to load
	Workflows []WorkflowConfig `yaml:"workflows,omitempty"`

	// ServiceClasses defines service class definitions to load
	ServiceClasses []ServiceClassConfig `yaml:"service_classes,omitempty"`
	// Services defines service instance definitions to load
	Services []ServiceConfig `yaml:"services,omitempty"`
	// MainConfig defines the main muster configuration
	MainConfig *MainConfig `yaml:"main_config,omitempty"`

	// MockOAuthServers defines mock OAuth servers to start for testing
	MockOAuthServers []MockOAuthServerConfig `yaml:"mock_oauth_servers,omitempty"`
}

MusterPreConfiguration defines how to pre-configure an muster serve instance

type RetryConfig 

type RetryConfig struct {
	// Count is the number of retry attempts
	Count int `yaml:"count"`
	// Delay between retry attempts
	Delay time.Duration `yaml:"delay"`
	// BackoffMultiplier for exponential backoff
	BackoffMultiplier float64 `yaml:"backoff_multiplier,omitempty"`
}

RetryConfig defines retry behavior for test steps

type ScenarioContext 

type ScenarioContext struct {
	// contains filtered or unexported fields
}

ScenarioContext holds the execution context for a test scenario including stored results from previous steps for template variable resolution

func NewScenarioContext 

func NewScenarioContext() *ScenarioContext

NewScenarioContext creates a new scenario execution context

func (*ScenarioContext) GetAllStoredResults 

func (sc *ScenarioContext) GetAllStoredResults() map[string]interface{}

GetAllStoredResults returns a copy of all stored results for debugging

func (*ScenarioContext) StoreResult 

func (sc *ScenarioContext) StoreResult(name string, result interface{})

StoreResult stores a step result under the given variable name

type ScenarioState 

type ScenarioState struct {
	Scenario    TestScenario     `json:"scenario"`
	StartTime   time.Time        `json:"start_time"`
	StepResults []TestStepResult `json:"step_results"`
	Status      string           `json:"status"` // "running", "completed", "failed"
}

ScenarioState tracks the state of a running scenario

type ScenarioValidationResult 

type ScenarioValidationResult struct {
	ScenarioName string                 `json:"scenario_name"`
	Valid        bool                   `json:"valid"`
	Errors       []ValidationError      `json:"errors,omitempty"`
	StepResults  []StepValidationResult `json:"step_results"`
}

ScenarioValidationResult represents the validation result for a single scenario

type ScenarioValidationResults 

type ScenarioValidationResults struct {
	TotalScenarios    int                        `json:"total_scenarios"`
	ValidScenarios    int                        `json:"valid_scenarios"`
	TotalErrors       int                        `json:"total_errors"`
	ScenarioResults   []ScenarioValidationResult `json:"scenario_results"`
	ValidationSummary map[string]int             `json:"validation_summary"`
}

ScenarioValidationResults represents the results of validating multiple scenarios

func ValidateScenariosAgainstSchema 

func ValidateScenariosAgainstSchema(scenarios []TestScenario, schema map[string]interface{}, verbose, debug bool) *ScenarioValidationResults

ValidateScenariosAgainstSchema validates scenarios against the API schema

type ServiceClassConfig 

type ServiceClassConfig map[string]interface{}

ServiceClassConfig represents a service class configuration

type ServiceConfig 

type ServiceConfig struct {
	// Name is the unique identifier for the service instance
	Name string `yaml:"name"`
	// Config contains the service instance definition
	Config map[string]interface{} `yaml:"config"`
}

ServiceConfig represents a service instance configuration

type StepValidationResult 

type StepValidationResult struct {
	StepID string            `json:"step_id"`
	Tool   string            `json:"tool"`
	Valid  bool              `json:"valid"`
	Errors []ValidationError `json:"errors,omitempty"`
}

StepValidationResult represents the validation result for a single step

type StructuredTestReporter 

type StructuredTestReporter interface {
	TestReporter
	// GetCurrentSuiteResult returns the current test suite result
	GetCurrentSuiteResult() *TestSuiteResult
	// GetScenarioStates returns the current state of all scenarios
	GetScenarioStates() map[string]*ScenarioState
	// GetCurrentResults returns the current scenario results
	GetCurrentResults() []TestScenarioResult
	// GetResultsAsJSON returns the current results as JSON
	GetResultsAsJSON() (string, error)
	// IsVerbose returns whether verbose reporting is enabled
	IsVerbose() bool
	// IsDebug returns whether debug reporting is enabled
	IsDebug() bool
}

StructuredTestReporter extends TestReporter with methods for structured data access This is typically used in MCP server mode where results need to be queried programmatically

type TemplateProcessor 

type TemplateProcessor struct {
	// contains filtered or unexported fields
}

TemplateProcessor handles template variable resolution in test step arguments using the existing enhanced template engine

func NewTemplateProcessor 

func NewTemplateProcessor(context *ScenarioContext) *TemplateProcessor

NewTemplateProcessor creates a new template processor with the given scenario context

func (*TemplateProcessor) ResolveArgs 

func (tp *TemplateProcessor) ResolveArgs(args map[string]interface{}) (map[string]interface{}, error)

ResolveArgs processes a map of arguments and resolves any template variables

type TestCategory 

type TestCategory string

TestCategory represents the category of tests to execute 

const (
	// CategoryBehavioral represents BDD-style behavioral tests
	CategoryBehavioral TestCategory = "behavioral"
	// CategoryIntegration represents integration and end-to-end tests
	CategoryIntegration TestCategory = "integration"
)

type TestConcept 

type TestConcept string

TestConcept represents the core muster concept being tested 

const (
	// ConceptServiceClass represents ServiceClass management tests
	ConceptServiceClass TestConcept = "serviceclass"
	// ConceptWorkflow represents Workflow execution tests
	ConceptWorkflow TestConcept = "workflow"
	// ConceptMCPServer represents MCPServer management tests
	ConceptMCPServer TestConcept = "mcpserver"

	// ConceptService represents Service lifecycle tests
	ConceptService TestConcept = "service"
)

type TestConfiguration 

type TestConfiguration struct {
	// Timeout is the overall test execution timeout
	Timeout time.Duration `yaml:"timeout"`
	// Category filter for test execution
	Category TestCategory `yaml:"category,omitempty"`
	// Concept filter for test execution
	Concept TestConcept `yaml:"concept,omitempty"`
	// Scenario filter for specific scenario execution
	Scenario string `yaml:"scenario,omitempty"`
	// Parallel is the number of parallel test workers
	Parallel int `yaml:"parallel"`
	// FailFast stops execution on first failure
	FailFast bool `yaml:"fail_fast"`
	// Verbose enables detailed output
	Verbose bool `yaml:"verbose"`
	// Debug enables debug logging and MCP tracing
	Debug bool `yaml:"debug"`
	// ConfigPath is the path to test scenario definitions
	ConfigPath string `yaml:"config_path,omitempty"`
	// ReportPath is the path to save detailed test reports
	ReportPath string `yaml:"report_path,omitempty"`
	// BasePort is the starting port number for muster instances
	BasePort int `yaml:"base_port,omitempty"`
	// KeepTempConfig keeps temporary config directory after test execution
	KeepTempConfig bool `yaml:"keep_temp_config,omitempty"`
}

TestConfiguration defines the overall test execution configuration

type TestExpectation 

type TestExpectation struct {
	// Success indicates whether the tool call should succeed
	Success bool `yaml:"success"`
	// ErrorContains checks if error message contains specific text
	ErrorContains []string `yaml:"error_contains,omitempty"`
	// Contains checks if response contains specific text
	Contains []string `yaml:"contains,omitempty"`
	// NotContains checks if response does not contain specific text
	NotContains []string `yaml:"not_contains,omitempty"`
	// JSONPath allows checking specific JSON response fields
	JSONPath map[string]interface{} `yaml:"json_path,omitempty"`
	// StatusCode for HTTP-based expectations
	StatusCode int `yaml:"status_code,omitempty"`
	// WaitForState enables polling for state changes with timeout
	WaitForState time.Duration `yaml:"wait_for_state,omitempty"`
}

TestExpectation defines what result is expected from a test step

type TestFramework 

type TestFramework struct {
	Runner          TestRunner
	Client          MCPTestClient
	Loader          TestScenarioLoader
	Reporter        TestReporter
	InstanceManager MusterInstanceManager
	Logger          TestLogger
}

TestFramework holds all components needed for testing

func NewTestFrameworkForMode 

func NewTestFrameworkForMode(mode ExecutionMode, verbose, debug bool, basePort int, reportPath string, keepTempConfig bool) (*TestFramework, error)

NewTestFrameworkForMode creates a fully configured test framework for the specified execution mode

Execution Modes:

  • ExecutionModeCLI: Uses standard stdio output for reporting (compatible with existing behavior)
  • ExecutionModeMCPServer: Uses structured reporting that captures data without stdio output to avoid contaminating the MCP protocol stream. Results can be retrieved programmatically.

Note: In MCP server mode, verbose defaults to true and debug output is controlled by the silent logger to prevent stdio contamination.

func NewTestFrameworkWithConfig 

func NewTestFrameworkWithConfig(verbose, debug bool, basePort int, reportPath string, keepTempConfig bool) (*TestFramework, error)

NewTestFrameworkWithConfig creates a fully configured test framework with all configuration options

func (*TestFramework) Cleanup 

func (tf *TestFramework) Cleanup() error

Cleanup cleans up resources used by the test framework

type TestLogger 

type TestLogger interface {
	// Debug logs debug-level messages (only shown when debug=true)
	Debug(format string, args ...interface{})
	// Info logs info-level messages (shown when verbose=true or debug=true)
	Info(format string, args ...interface{})
	// Error logs error-level messages (always shown)
	Error(format string, args ...interface{})
	// IsDebugEnabled returns whether debug logging is enabled
	IsDebugEnabled() bool
	// IsVerboseEnabled returns whether verbose logging is enabled
	IsVerboseEnabled() bool
}

TestLogger provides centralized logging for test execution

func NewPrefixedLogger 

func NewPrefixedLogger(base TestLogger, prefix string) TestLogger

NewPrefixedLogger creates a new logger that wraps the base logger and adds a prefix to all log messages. The prefix is typically a short unique identifier for the scenario being executed.

func NewSilentLogger 

func NewSilentLogger(verbose, debug bool) TestLogger

NewSilentLogger creates a logger that suppresses all output (for MCP server mode)

func NewStdoutLogger 

func NewStdoutLogger(verbose, debug bool) TestLogger

NewStdoutLogger creates a logger that outputs to stdout/stderr

type TestReporter 

type TestReporter interface {
	// ReportStart is called when test execution begins
	ReportStart(config TestConfiguration)
	// ReportScenarioStart is called when a scenario begins
	ReportScenarioStart(scenario TestScenario)
	// ReportStepResult is called when a step completes
	ReportStepResult(stepResult TestStepResult)
	// ReportScenarioResult is called when a scenario completes
	ReportScenarioResult(scenarioResult TestScenarioResult)
	// ReportSuiteResult is called when all tests complete
	ReportSuiteResult(suiteResult TestSuiteResult)
	// SetParallelMode enables or disables parallel output buffering
	SetParallelMode(parallel bool)
}

TestReporter interface defines how test results are reported

func NewStructuredReporter 

func NewStructuredReporter(verbose, debug bool, reportPath string) TestReporter

NewStructuredReporter creates a reporter that captures structured data without stdio output

func NewTestReporter 

func NewTestReporter(verbose, debug bool, reportPath string) TestReporter

NewTestReporter creates a new test reporter

type TestResult 

type TestResult string

TestResult represents the result of test execution 

const (
	// ResultPassed indicates the test passed successfully
	ResultPassed TestResult = "PASSED"
	// ResultFailed indicates the test failed
	ResultFailed TestResult = "FAILED"
	// ResultSkipped indicates the test was skipped
	ResultSkipped TestResult = "SKIPPED"
	// ResultError indicates an error occurred during test execution
	ResultError TestResult = "ERROR"
)

type TestRunner 

type TestRunner interface {
	// Run executes test scenarios according to the configuration
	Run(ctx context.Context, config TestConfiguration, scenarios []TestScenario) (*TestSuiteResult, error)
}

TestRunner interface defines the test execution engine

func NewTestRunnerWithLogger 

func NewTestRunnerWithLogger(client MCPTestClient, loader TestScenarioLoader, reporter TestReporter, instanceManager MusterInstanceManager, debug bool, logger TestLogger) TestRunner

NewTestRunnerWithLogger creates a new test runner with custom logger

type TestScenario 

type TestScenario struct {
	// Name is the unique identifier for the scenario
	Name string `yaml:"name"`
	// Category is the test category (behavioral, integration)
	Category TestCategory `yaml:"category"`
	// Concept is the core concept being tested
	Concept TestConcept `yaml:"concept"`
	// Description provides human-readable scenario description
	Description string `yaml:"description"`
	// Prerequisites define setup requirements
	Prerequisites []string `yaml:"prerequisites,omitempty"`
	// Steps define the test execution steps
	Steps []TestStep `yaml:"steps"`
	// Cleanup defines teardown steps
	Cleanup []TestStep `yaml:"cleanup,omitempty"`
	// Timeout for this specific scenario
	Timeout time.Duration `yaml:"timeout,omitempty"`
	// Tags for additional categorization
	Tags []string `yaml:"tags,omitempty"`
	// Skip indicates whether this scenario should be skipped
	Skip bool `yaml:"skip,omitempty"`
	// PreConfiguration defines muster instance setup
	PreConfiguration *MusterPreConfiguration `yaml:"pre_configuration,omitempty"`
}

TestScenario defines a single test scenario

func LoadAndFilterScenarios 

func LoadAndFilterScenarios(configPath string, config TestConfiguration, logger TestLogger) ([]TestScenario, error)

LoadAndFilterScenarios provides a unified way to load and filter scenarios

func LoadScenariosForCompletion 

func LoadScenariosForCompletion(configPath string) ([]TestScenario, error)

LoadScenariosForCompletion provides a simple way to load scenarios for shell completion This uses minimal logging to avoid interfering with completion output

type TestScenarioLoader 

type TestScenarioLoader interface {
	// LoadScenarios loads test scenarios from the given path
	LoadScenarios(configPath string) ([]TestScenario, error)
	// FilterScenarios filters scenarios based on the configuration
	FilterScenarios(scenarios []TestScenario, config TestConfiguration) []TestScenario
}

TestScenarioLoader interface defines how test scenarios are loaded

func CreateScenarioLoaderForContext 

func CreateScenarioLoaderForContext(debug bool, logger TestLogger) TestScenarioLoader

CreateScenarioLoaderForContext creates a scenario loader appropriate for the given context

func NewTestScenarioLoader 

func NewTestScenarioLoader(debug bool) TestScenarioLoader

NewTestScenarioLoader creates a new test scenario loader

func NewTestScenarioLoaderWithLogger 

func NewTestScenarioLoaderWithLogger(debug bool, logger TestLogger) TestScenarioLoader

NewTestScenarioLoaderWithLogger creates a new test scenario loader with custom logger

type TestScenarioResult 

type TestScenarioResult struct {
	// Scenario is the scenario that was executed
	Scenario TestScenario `json:"scenario"`
	// Result is the overall result of the scenario
	Result TestResult `json:"result"`
	// StartTime when scenario execution began
	StartTime time.Time `json:"start_time"`
	// EndTime when scenario execution completed
	EndTime time.Time `json:"end_time"`
	// Duration of scenario execution
	Duration time.Duration `json:"duration"`
	// StepResults contains individual step results
	StepResults []TestStepResult `json:"step_results"`
	// Error message if the scenario failed or had an error
	Error string `json:"error,omitempty"`
	// Output from scenario execution
	Output string `json:"output,omitempty"`
	// InstanceLogs contains logs from the muster serve instance
	InstanceLogs *InstanceLogs `json:"instance_logs,omitempty"`
}

TestScenarioResult represents the result of a single test scenario

type TestStep 

type TestStep struct {
	// ID is the step identifier
	ID string `yaml:"id"`
	// Description explains what the step does
	Description string `yaml:"description,omitempty"`
	// Tool is the MCP tool to invoke
	Tool string `yaml:"tool"`
	// Args are the tool arguments as a map
	Args map[string]interface{} `yaml:"args"`
	// Expected defines the expected outcome
	Expected TestExpectation `yaml:"expected"`
	// Retry configuration for this step
	Retry *RetryConfig `yaml:"retry,omitempty"`
	// Timeout for this specific step
	Timeout time.Duration `yaml:"timeout,omitempty"`
	// AsUser specifies which user session to execute this step as.
	// For multi-user testing scenarios. If not set, uses the current user.
	AsUser string `yaml:"as_user,omitempty"`
}

TestStep defines a single step within a test scenario

type TestStepResult 

type TestStepResult struct {
	// Step is the step that was executed
	Step TestStep `json:"step"`
	// Result is the result of the step
	Result TestResult `json:"result"`
	// StartTime when step execution began
	StartTime time.Time `json:"start_time"`
	// EndTime when step execution completed
	EndTime time.Time `json:"end_time"`
	// Duration of step execution
	Duration time.Duration `json:"duration"`
	// Response from the MCP tool call
	Response interface{} `json:"response,omitempty"`
	// Error message if the step failed
	Error string `json:"error,omitempty"`
	// RetryCount is the number of retries attempted
	RetryCount int `json:"retry_count"`
}

TestStepResult represents the result of a single test step

type TestSuiteResult 

type TestSuiteResult struct {
	// StartTime when test execution began
	StartTime time.Time `json:"start_time"`
	// EndTime when test execution completed
	EndTime time.Time `json:"end_time"`
	// Duration of test execution
	Duration time.Duration `json:"duration"`
	// TotalScenarios is the total number of scenarios executed
	TotalScenarios int `json:"total_scenarios"`
	// PassedScenarios is the number of scenarios that passed
	PassedScenarios int `json:"passed_scenarios"`
	// FailedScenarios is the number of scenarios that failed
	FailedScenarios int `json:"failed_scenarios"`
	// SkippedScenarios is the number of scenarios that were skipped
	SkippedScenarios int `json:"skipped_scenarios"`
	// ErrorScenarios is the number of scenarios that had errors
	ErrorScenarios int `json:"error_scenarios"`
	// ScenarioResults contains individual scenario results
	ScenarioResults []TestScenarioResult `json:"scenario_results"`
	// Configuration used for this test run
	Configuration TestConfiguration `json:"configuration"`
}

TestSuiteResult represents the overall result of test suite execution

type TestToolContent 

type TestToolContent struct {
	Type string `json:"type"`
	Text string `json:"text"`
}

TestToolContent represents content in a test tool result.

type TestToolResult 

type TestToolResult struct {
	Content []TestToolContent `json:"content"`
	IsError bool              `json:"isError,omitempty"`
}

TestToolResult wraps a result from a test tool to match MCP response format.

func WrapTestToolResult 

func WrapTestToolResult(result interface{}, err error) *TestToolResult

WrapTestToolResult wraps a test tool result in MCP-compatible format.

type TestToolsHandler 

type TestToolsHandler struct {
	// contains filtered or unexported fields
}

TestToolsHandler handles test-specific tools that operate on mock infrastructure. These tools are NOT exposed through the muster serve MCP server - they are handled directly by the test runner before delegating to the real MCP client.

func NewTestToolsHandler 

func NewTestToolsHandler(instanceManager MusterInstanceManager, instance *MusterInstance, debug bool, logger TestLogger) *TestToolsHandler

NewTestToolsHandler creates a new test tools handler.

func (*TestToolsHandler) CloseAllUserClients 

func (h *TestToolsHandler) CloseAllUserClients()

CloseAllUserClients closes all user MCP clients except the default one. The default client is managed by the test runner.

func (*TestToolsHandler) GetCurrentClient 

func (h *TestToolsHandler) GetCurrentClient() MCPTestClient

GetCurrentClient returns the MCP client for the currently active user.

func (*TestToolsHandler) GetCurrentUserName 

func (h *TestToolsHandler) GetCurrentUserName() string

GetCurrentUserName returns the name of the currently active user.

func (*TestToolsHandler) HandleTestTool 

func (h *TestToolsHandler) HandleTestTool(ctx context.Context, toolName string, args map[string]interface{}) (interface{}, error)

HandleTestTool executes a test helper tool and returns the result.

func (*TestToolsHandler) HasUser 

func (h *TestToolsHandler) HasUser(name string) bool

HasUser returns true if a user session with the given name exists. This provides encapsulated access to check user existence without exposing the internal userClients map.

func (*TestToolsHandler) SetMCPClient 

func (h *TestToolsHandler) SetMCPClient(client MCPTestClient)

SetMCPClient sets the MCP client for calling tools in the muster instance. This is used by test_simulate_oauth_callback to call the authenticate tool.

func (*TestToolsHandler) SwitchToUser 

func (h *TestToolsHandler) SwitchToUser(name string) bool

SwitchToUser switches the current user context to the specified user. This is used by the test runner to switch user context for as_user steps. Returns false if the user doesn't exist.

type TrustedIssuerConfig 

type TrustedIssuerConfig struct {
	// ConnectorID is the identifier used in the token exchange request
	// to specify which connector to use (maps to audience in RFC 8693)
	ConnectorID string `yaml:"connector_id"`

	// OAuthServerRef references the mock OAuth server by name whose tokens
	// should be accepted for exchange
	OAuthServerRef string `yaml:"oauth_server_ref"`
}

TrustedIssuerConfig defines a trusted issuer for RFC 8693 token exchange

type ValidationError 

type ValidationError struct {
	Type       string `json:"type"`
	Message    string `json:"message"`
	Field      string `json:"field,omitempty"`
	Suggestion string `json:"suggestion,omitempty"`
}

ValidationError represents a validation error

type WorkflowConfig 

type WorkflowConfig struct {
	// Name is the unique identifier for the workflow
	Name string `yaml:"name"`
	// Config contains the workflow definition
	Config map[string]interface{} `yaml:"config"`
}

WorkflowConfig represents a workflow configuration

Source Files

View all Source files

Directories

Path Synopsis
fixtures
oauth
Package oauth contains test fixtures for OAuth authentication testing.
 Package oauth contains test fixtures for OAuth authentication testing.
Package mock provides mock MCP server functionality for testing muster components.
 Package mock provides mock MCP server functionality for testing muster components.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.