engine

package
v1.5.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 21, 2026 License: Apache-2.0 Imports: 76 Imported by: 0

Documentation

Overview

Package engine orchestrates test execution across scenarios, providers, and configurations.

The engine package is the core execution layer of the Arena testing tool. It manages:

  • Conversation lifecycle and message flow
  • Provider and model configuration
  • Telemetry and metrics collection
  • Result aggregation and validation
  • Concurrent test execution across multiple scenarios

Key types:

  • Engine: Main orchestration struct that executes test runs
  • RunResult: Contains execution results, metrics, and conversation history
  • RunFilters: Filters for selective test execution

Example usage: 

eng, _ := engine.NewEngine(cfg, providerRegistry, promptRegistry)
results, _ := eng.Execute(ctx, filters)
for _, result := range results {
    fmt.Printf("Scenario %s: %s\n", result.ScenarioID, result.Status)
}

Index

Constants

View Source 
const DefaultRunTimeout = 5 * time.Minute

DefaultRunTimeout is the default maximum duration for a single run execution. If a provider call hangs, the run will be canceled after this timeout, freeing up the semaphore slot for other runs.

Variables

This section is empty.

Functions

func AggregateTrialResults 

func AggregateTrialResults(store *statestore.ArenaStateStore, runIDs []string, combos []RunCombination) []string

AggregateTrialResults groups trial run results by scenario+provider+region, computes statistical metrics, and updates the first run in each group with the aggregated TrialResults. Returns the run IDs that represent trial groups (i.e., the first run ID of each group that now carries the summary).

func ApplyPerturbation 

func ApplyPerturbation(turns []arenaconfig.TurnDefinition, variant PerturbationVariant) []arenaconfig.TurnDefinition

ApplyPerturbation substitutes perturbation variables in turn content. Placeholders use {key} syntax (e.g., "Book a flight from {city}" with city=NYC becomes "Book a flight from NYC").

func FormatFileSize 

func FormatFileSize(bytes int64) string

FormatFileSize formats bytes as human-readable size

func FormatMediaType 

func FormatMediaType(mediaType string) string

FormatMediaType returns a human-readable label for media type

Types

type A2AAgentInfo 

type A2AAgentInfo struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Skills      []A2ASkillInfo `json:"skills,omitempty"`
}

A2AAgentInfo contains metadata about an A2A agent for report rendering.

type A2ASkillInfo 

type A2ASkillInfo struct {
	ID          string   `json:"id"`
	Name        string   `json:"name"`
	Description string   `json:"description,omitempty"`
	Tags        []string `json:"tags,omitempty"`
}

A2ASkillInfo contains metadata about a single A2A agent skill.

type AgentInfo 

type AgentInfo struct {
	TaskType    string
	Description string
}

AgentInfo describes one selectable agent in an Arena config. An "agent" is a prompt config, keyed by its task_type, carrying the system prompt and tools.

type AssertionTrialStats 

type AssertionTrialStats struct {
	// PassRate is the fraction of trials where this assertion passed.
	PassRate float64 `json:"pass_rate"`
	// PassCount is the number of trials where this assertion passed.
	PassCount int `json:"pass_count"`
	// FailCount is the number of trials where this assertion failed.
	FailCount int `json:"fail_count"`
	// FlakinessScore for this specific assertion.
	FlakinessScore float64 `json:"flakiness_score"`
}

AssertionTrialStats holds per-assertion statistics across trial runs.

type AssertionsSummary 

type AssertionsSummary struct {
	Failed  int                                       `json:"failed"`
	Passed  bool                                      `json:"passed"`
	Results []assertions.ConversationValidationResult `json:"results"`
	Total   int                                       `json:"total"`
}

AssertionsSummary matches the structure used for turn-level assertions in message meta

type AudioMonitorHook 

type AudioMonitorHook func(runID string, router *arenaaudio.AudioRouter, rate int)

AudioMonitorHook is invoked when the engine constructs a per-run AudioRouter. Hooks can use the router to attach additional consumers (web SSE relay, TUI level meter). The router lifecycle is the run; hooks must not retain the pointer after the corresponding run completes and must not call Close (the engine owns lifecycle).

runID is the unique identifier for this run; rate is the canonical sample rate of the router.

type CompositeConversationExecutor 

type CompositeConversationExecutor struct {
	// contains filtered or unexported fields
}

CompositeConversationExecutor routes conversation execution to the appropriate executor based on scenario configuration. It selects between: - EvalConversationExecutor for evaluation mode (recording replay with assertions) - DefaultConversationExecutor for standard turn-based conversations - DuplexConversationExecutor for bidirectional streaming scenarios

func NewCompositeConversationExecutor 

func NewCompositeConversationExecutor(
	defaultExecutor *DefaultConversationExecutor,
	duplexExecutor *DuplexConversationExecutor,
	evalExecutor *EvalConversationExecutor,
) *CompositeConversationExecutor

NewCompositeConversationExecutor creates a new composite executor.

func (*CompositeConversationExecutor) ExecuteConversation 

func (ce *CompositeConversationExecutor) ExecuteConversation(
	ctx context.Context,
	req ConversationRequest,
) *ConversationResult

ExecuteConversation routes to the appropriate executor based on scenario config. If the request is for eval mode, uses EvalConversationExecutor. If the scenario has duplex configuration, uses DuplexConversationExecutor. Otherwise, uses DefaultConversationExecutor.

func (*CompositeConversationExecutor) ExecuteConversationStream 

func (ce *CompositeConversationExecutor) ExecuteConversationStream(
	ctx context.Context,
	req ConversationRequest,
) (<-chan ConversationStreamChunk, error)

ExecuteConversationStream routes streaming execution to the appropriate executor.

func (*CompositeConversationExecutor) GetDefaultExecutor 

func (ce *CompositeConversationExecutor) GetDefaultExecutor() *DefaultConversationExecutor

GetDefaultExecutor returns the default executor for direct access if needed.

func (*CompositeConversationExecutor) GetDuplexExecutor 

func (ce *CompositeConversationExecutor) GetDuplexExecutor() *DuplexConversationExecutor

GetDuplexExecutor returns the duplex executor for direct access if needed.

func (*CompositeConversationExecutor) GetEvalExecutor 

func (ce *CompositeConversationExecutor) GetEvalExecutor() *EvalConversationExecutor

GetEvalExecutor returns the eval executor for direct access if needed.

type CompositionMetadataProvider 

type CompositionMetadataProvider interface {
	CompositionMetadata() map[string]any
}

CompositionMetadataProvider is implemented by types that provide per-turn composition observability (step outputs, branch-taken, parallel status) for injection into the eval context, so composition_* assertions can target intermediate steps. Set per-run for composition scenarios (RFC 0010).

type ConversationExecutor 

type ConversationExecutor interface {
	// ExecuteConversation runs a complete conversation based on scenario
	ExecuteConversation(ctx context.Context, req ConversationRequest) *ConversationResult

	// ExecuteConversationStream runs a conversation with streaming
	ExecuteConversationStream(ctx context.Context, req ConversationRequest) (<-chan ConversationStreamChunk, error)
}

ConversationExecutor orchestrates full conversation flows

func BuildEngineComponents 

func BuildEngineComponents(cfg *arenaconfig.Config, providerFilter []string) (
	providerRegistry *providers.Registry,
	promptRegistry *prompt.Registry,
	mcpRegistry *mcp.RegistryImpl,
	convExecutor ConversationExecutor,
	adapterReg *adapters.Registry,
	a2aCleanup func(),
	toolReg *tools.Registry,
	skillExec *skills.Executor,
	err error,
)

BuildEngineComponents builds all engine components from a loaded Config object. This function creates and initializes: - MCP registry and tools (if configured) - Provider registry (for main assistant) - Prompt registry (if configured) - Tool registry (static + MCP tools) - Turn executor - Self-play provider registry (if enabled) - Self-play registry (if enabled) - Conversation executor

This function is exported to enable programmatic creation of Arena engines without requiring file-based configuration. Users can construct a *arenaconfig.Config programmatically and pass it to this function to get all required registries for use with NewEngine.

Returns all components needed to construct an Engine, or an error if any component fails to build.

type ConversationRequest 

type ConversationRequest struct {
	// Required fields
	Provider providers.Provider
	Scenario *arenaconfig.Scenario
	Eval     *arenaconfig.Eval // Eval configuration (mutually exclusive with Scenario)
	Config   *arenaconfig.Config
	Region   string

	// Optional overrides (for future use)
	Temperature *float64 // Override scenario temperature
	MaxTokens   *int     // Override scenario max tokens
	Timeout     *int     // Timeout in seconds

	// For distributed execution and tracing (v0.2.0+)
	RunID    string            // Unique identifier for this run
	Metadata map[string]string // Additional metadata for debugging/tracing

	// Event bus for runtime/TUI events
	EventBus events.Bus

	// WorkflowStateResolver lets the provider tool loop hand a turn over to a
	// workflow state mid-flight: it commits the pending transition and swaps in
	// the destination state's prompt and tools between rounds. Nil for
	// non-workflow runs, which the runtime treats as "no workflow" and skips.
	WorkflowStateResolver stage.WorkflowStateResolver

	// State management
	StateStoreConfig *StateStoreConfig // Optional state store configuration
	ConversationID   string            // Conversation identifier for state persistence

	// Per-run eval orchestrator override (for workflow scenarios that need
	// isolated workflow metadata). If nil, the executor's shared orchestrator is used.
	EvalOrchestrator *EvalOrchestrator

	// VoiceSTT is the STT provider for the VAD voice path (Task 7). Nil in ASM mode.
	// Set from --voice-stt by the chat command; consumed by runInteractiveVADVoice.
	VoiceSTT *config.Provider

	// VoiceOutputVoice is the TTS voice ID for the VAD voice path (Task 7).
	// Set from --voice-output-voice by the chat command; consumed by runInteractiveVADVoice.
	VoiceOutputVoice string

	// VoiceBargeIn enables barge-in (interrupting the agent mid-reply). On the
	// composed-VAD path it shares an InterruptionHandler across AudioTurn + TTS;
	// on the ASM/realtime path it watches the session's out-of-band BargeIn()
	// channel and flushes playback. Opt-in (--barge-in); off by default the
	// console does clean turn-taking.
	VoiceBargeIn bool

	// OnDuplexSession, when set, is forwarded to the DuplexProviderStage as its
	// session observer (SetSessionObserver) — invoked once with the streaming
	// session right after it is created. The interactive ASM path uses it to wire
	// barge-in (session.BargeIn → playback flush). Nil for non-interactive runs.
	OnDuplexSession func(providers.StreamInputSession)

	// RecordingConfig enables RecordingStage in the pipeline for message.created events.
	// If nil, no recording stages are added.
	RecordingConfig *stage.RecordingStageConfig

	// EventStore is the destination for RecordingStage writes. Required when
	// RecordingConfig is non-nil; without it, recording stages will not be added
	// to the pipeline.
	EventStore events.EventStore

	// PostTurnHook is called after each turn completes. Used by the workflow
	// engine to commit deferred transitions after the pipeline finishes.
	PostTurnHook func() error

	// ContextEnricher is called before each turn to enrich the pipeline context.
	// Used to inject per-run state (e.g., skill filters) into context for tool execution.
	ContextEnricher func(ctx context.Context) context.Context

	// ActiveCompositionResolver, when non-nil, is called per-turn to determine
	// the active composition for the current workflow state (RFC 0010). It returns
	// nil for states that are not composition-orchestrated.
	ActiveCompositionResolver func() *composition.Composition

	// CompositionRecorder is the per-run recorder for RFC 0010 testability. When
	// non-nil it is stamped onto every TurnRequest so buildStagePipeline can pass
	// it to NewCompositionStageWithRecorder. Reset() is called per turn so stale
	// data from a prior turn does not leak into the next turn's assertions.
	CompositionRecorder *stage.CompositionRecorder

	// AudioRouter, when non-nil, is the per-run AudioRouter for audio
	// monitoring. MonitorTap stages added to the pipeline publish to this
	// router. Lifetime is the run; the engine owns Close().
	AudioRouter *arenaaudio.AudioRouter

	// CurrentWorkflowState returns the live workflow-state metadata for the turn
	// about to run (captured at turn start, before the pipeline builds its
	// prompt). Nil when there is no workflow. Used to stamp current_workflow_state
	// onto each assistant result so composition-orchestrated terminal states —
	// which leave no transition tool-result message — are still visible per turn.
	CurrentWorkflowState func() map[string]interface{}

	// StampWorkflowState attaches meta (captured at turn start) to the assistant
	// message produced by the just-completed turn. No-op when meta is nil. This is
	// side-effect-only and must never alter turn output, error, or flow.
	StampWorkflowState func(meta map[string]interface{})
	// contains filtered or unexported fields
}

ConversationRequest contains all data needed for conversation execution. Using a request object makes the API extensible without breaking changes.

type ConversationResult 

type ConversationResult struct {
	Messages     []types.Message         // Flat list of all messages in the conversation
	Cost         types.CostInfo          // Total cost across all messages
	ToolStats    *types.ToolStats        // Tool usage statistics
	Violations   []types.ValidationError // Validation errors
	MediaOutputs []MediaOutput           // Media outputs generated by LLMs

	// Conversation-level assertions (test-only gates: passed/failed)
	ConversationAssertionResults []assertions.ConversationValidationResult `json:"conv_assertions_results,omitempty"`

	// EvalResults are the runtime's pack-level eval observations —
	// non-gating measurements that fire during a session (the same
	// thing that would surface in production). Distinct from
	// ConversationAssertionResults because evals don't pass/fail;
	// they emit structured Value/Details for reporting and aggregation.
	EvalResults []evals.EvalResult `json:"eval_results,omitempty"`

	// Self-play metadata
	SelfPlay  bool   `json:"self_play,omitempty"`
	PersonaID string `json:"persona_id,omitempty"`

	// Error handling
	Error  string `json:"error,omitempty"`  // Error message if execution failed
	Failed bool   `json:"failed,omitempty"` // Whether execution failed (but partial results may be available)

	// Err is the underlying typed error behind Error, preserved (not
	// serialized) so retry logic can classify by type/status instead of
	// matching on the stringified message. nil unless Failed.
	Err error `json:"-"`

	Skipped    bool   `json:"skipped,omitempty"`     // Whether execution was skipped due to transient provider error
	SkipReason string `json:"skip_reason,omitempty"` // Reason for skipping
}

ConversationResult contains the outcome of conversation execution

type ConversationStreamChunk 

type ConversationStreamChunk struct {
	// Current turn number (0-indexed)
	TurnIndex int

	// Delta content from this specific chunk
	Delta string

	// Token count (accumulated for current turn)
	TokenCount int

	// Finish reason for current turn (only in last chunk of turn)
	FinishReason *string

	// Complete conversation result (accumulated, updated with each chunk)
	Result *ConversationResult

	// Error if streaming failed
	Error error

	// Metadata
	Metadata map[string]interface{}
}

ConversationStreamChunk represents a streaming chunk during conversation execution

type DefaultConversationExecutor 

type DefaultConversationExecutor struct {
	// contains filtered or unexported fields
}

DefaultConversationExecutor implements ConversationExecutor interface

func NewDefaultConversationExecutor 

func NewDefaultConversationExecutor(
	scriptedExecutor turnexecutors.TurnExecutor,
	selfPlayExecutor turnexecutors.TurnExecutor,
	selfPlayRegistry *selfplay.Registry,
	promptRegistry *prompt.Registry,
	evalOrchestrator *EvalOrchestrator,
) *DefaultConversationExecutor

NewDefaultConversationExecutor creates a new conversation executor

func (*DefaultConversationExecutor) ExecuteConversation 

func (ce *DefaultConversationExecutor) ExecuteConversation(ctx context.Context, req ConversationRequest) *ConversationResult

ExecuteConversation runs a complete conversation based on scenario using the new Turn model

func (*DefaultConversationExecutor) ExecuteConversationStream 

func (ce *DefaultConversationExecutor) ExecuteConversationStream(ctx context.Context, req ConversationRequest) (<-chan ConversationStreamChunk, error)

ExecuteConversationStream runs a complete conversation with streaming

type DuplexConversationExecutor 

type DuplexConversationExecutor struct {
	// contains filtered or unexported fields
}

DuplexConversationExecutor handles duplex (bidirectional streaming) conversations. Unlike the standard executor which processes turns sequentially, this executor establishes a persistent streaming session and handles real-time audio I/O.

func NewDuplexConversationExecutor 

func NewDuplexConversationExecutor(
	selfPlayRegistry *selfplay.Registry,
	promptRegistry *prompt.Registry,
	toolRegistry *tools.Registry,
	mediaStorage storage.MediaStorageService,
	evalOrchestrator *EvalOrchestrator,
) *DuplexConversationExecutor

NewDuplexConversationExecutor creates a new duplex conversation executor.

func (*DuplexConversationExecutor) ExecuteConversation 

func (de *DuplexConversationExecutor) ExecuteConversation(
	ctx context.Context,
	req ConversationRequest,
) *ConversationResult

ExecuteConversation runs a duplex conversation based on scenario. For duplex mode, this establishes a streaming session and processes audio turns in real-time.

func (*DuplexConversationExecutor) ExecuteConversationStream 

func (de *DuplexConversationExecutor) ExecuteConversationStream(
	ctx context.Context,
	req ConversationRequest,
) (<-chan ConversationStreamChunk, error)

ExecuteConversationStream runs a duplex conversation with streaming output. For duplex mode, this returns chunks as they arrive from the provider.

func (*DuplexConversationExecutor) RunInteractiveVoice 

func (de *DuplexConversationExecutor) RunInteractiveVoice(
	ctx context.Context,
	req *ConversationRequest,
	mic <-chan []byte,
	play func([]byte),
	flush func(),
) error

RunInteractiveVoice drives a live, mic-fed voice conversation through the duplex pipeline until mic is closed or ctx is canceled.

Mode selection (in priority order):

  1. req.VoiceSTT != nil → composed VAD pipeline (AudioTurn → STT → LLM → TTS). The caller explicitly configured an STT provider via --voice-stt, signaling the composed path even when the underlying provider also implements StreamInputSupport. The OpenAI provider implements StreamInputSupport unconditionally for every model (including plain text models like gpt-4o), so checking VoiceSTT first prevents a plain-text agent from being wrongly routed to the ASM/realtime session path (which would attempt to create a gpt-4o-realtime-preview session and fail with model_not_found).
  2. provider implements StreamInputSupport → ASM pipeline (buildDuplexPipeline → DuplexProviderStage → ArenaStateStoreSaveStage).
  3. fallback → composed VAD pipeline (runInteractiveVADVoice will return an error if VoiceSTT is nil, explaining the missing --voice-stt flag).

Mic frames (raw PCM16 mono @ 16 kHz or provider-preferred rate) are forwarded to the pipeline input channel one element per frame. Response audio from the pipeline output channel is delivered to play concurrently; the drain goroutine is joined before RunInteractiveVoice returns so every response frame is delivered to play by the time the function exits.

When mic is closed, an EndOfStream element is sent to the pipeline to signal end-of-user-speech to the provider (matching how processSingleDuplexTurn does it), then inputChan is closed and the output drain is awaited.

History (transcripts + tool calls) is persisted by ArenaStateStoreSaveStage inside the pipeline, exactly as for a duplex scenario run.

func (*DuplexConversationExecutor) RunRealtimeSession added in v1.5.7 

func (de *DuplexConversationExecutor) RunRealtimeSession(
	ctx context.Context,
	req *ConversationRequest,
	sess audio.Session,
) error

RunRealtimeSession drives a live duplex (voice) conversation using an externally supplied audio.Session for capture and playback, instead of the process-local microphone and speaker. It lets a library consumer bridge any audio transport — e.g. a WebRTC track — into a headless realtime conversation: implement audio.Session (one Source for capture, one Sink for playback) and pass it here.

Contract:

  • The Session's first Source must emit PCM16 mono @ 16 kHz capture frames. Its Frames channel closing (or ctx being canceled) ends the conversation, signaling end-of-user-speech to the provider.
  • Response audio is written to the Session's first Sink as PCM16 mono frames stamped at 24 kHz; Flush is called on barge-in to drop in-flight playback.
  • Session.Start must begin capture/playback and return (non-blocking), like PortAudio's; the Session is Closed before this method returns.

This entrypoint pulls in only the pure runtime/audio interfaces — it does not import the PortAudio device binding — so a binary that uses it (and avoids the voice/TUI packages) stays CGO_ENABLED=0 / statically linkable for scratch containers. See RunInteractiveVoice for the underlying channel-based seam.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine manages the execution of prompt testing scenarios across multiple providers, regions, and configurations. It coordinates conversation execution, tool calling, validation, and result collection.

The engine supports both scripted conversations and self-play mode where an LLM simulates user behavior. It handles provider initialization, concurrent execution, and comprehensive result tracking including costs and tool usage.

func NewEngine 

func NewEngine(
	cfg *arenaconfig.Config,
	providerRegistry *providers.Registry,
	promptRegistry *prompt.Registry,
	mcpRegistry *mcp.RegistryImpl,
	convExecutor ConversationExecutor,
	adapterRegistry *adapters.Registry,
	toolRegistry *tools.Registry,
) (*Engine, error)

NewEngine creates a new simulation engine from pre-built components. This is the primary constructor for the Engine and is preferred for testing where components can be created and configured independently.

This constructor uses dependency injection, accepting all required registries and executors as parameters. This makes testing easier and follows better architectural practices.

Parameters:

  • cfg: Fully loaded and validated Config object
  • providerRegistry: Registry for looking up providers by ID
  • promptRegistry: Registry for system prompts and task types
  • convExecutor: Executor for full conversations
  • adapterRegistry: Registry for recording adapters (used for eval enumeration)

Returns an initialized Engine ready for test execution.

func NewEngineFromConfig 

func NewEngineFromConfig(cfg *arenaconfig.Config, providerFilter ...string) (*Engine, error)

NewEngineFromConfig creates a new Engine from a pre-loaded configuration. This allows CLI or programmatic callers to modify the config before engine creation. providerFilter limits which providers have credentials resolved; nil or empty means all providers are initialized.

func NewEngineFromConfigFile 

func NewEngineFromConfigFile(configPath string) (*Engine, error)

NewEngineFromConfigFile creates a new simulation engine from a configuration file. It loads the configuration, validates it, initializes all registries, and sets up the execution pipeline for conversation testing.

The configuration file is loaded along with all referenced resources (scenarios, providers, tools, personas), making the Config object fully self-contained.

This constructor performs all necessary initialization steps in the correct order: 1. Load and validate configuration from file (including all referenced resources) 2. Build registries from loaded resources 3. Initialize executors (turn, conversation, self-play if enabled) 4. Create Engine with all components

Note: Logger verbosity should be configured at application startup, not here. This function does not modify global logger settings.

Parameters:

  • configPath: Path to the arena.yaml configuration file

Returns an initialized Engine ready for test execution, or an error if:

  • Configuration file cannot be read or parsed
  • Configuration validation fails
  • Any resource file cannot be loaded
  • Provider type is unsupported

func (*Engine) Agents 

func (e *Engine) Agents() []AgentInfo

Agents lists the prompt configs (agents) declared in the loaded config, sorted by task_type for stable ordering in pickers.

func (*Engine) AudioMonitorEnabled 

func (e *Engine) AudioMonitorEnabled() bool

AudioMonitorEnabled reports whether EnableAudioMonitor was called. Callers use this to decide whether to construct an arenaaudio.Monitor.

func (*Engine) AudioMonitorOptions 

func (e *Engine) AudioMonitorOptions() arenaaudio.Options

AudioMonitorOptions returns the configured monitor options. Zero value when monitoring is disabled — check AudioMonitorEnabled first.

func (*Engine) Close 

func (e *Engine) Close() error

Close shuts down the engine and cleans up resources. This includes closing all MCP server connections, provider HTTP clients, and the event store if session recording is enabled.

func (*Engine) ConfigureSessionRecordingFromConfig 

func (e *Engine) ConfigureSessionRecordingFromConfig() error

ConfigureSessionRecordingFromConfig enables session recording if configured. It reads the recording configuration from the engine's config and enables session recording with the appropriate directory path. Returns nil if recording is not enabled in the config.

func (*Engine) EnableAudioMonitor 

func (e *Engine) EnableAudioMonitor(opts arenaaudio.Options) error

EnableAudioMonitor enables real-time audio monitoring for duplex scenarios.

The monitor surfaces (LocalSink, SSE relay, TUI level meter) are added to per-run pipelines only when at least one consumer is wired and the scenario uses duplex streaming. CI / non-interactive runs (auto mode + no TTY) do not construct any audio infrastructure.

Options.Rate must be one of audio.Rate16k, Rate24k, Rate48k.

This method only stores configuration; the per-run wiring happens in the pipeline construction path (see duplex_executor_pipeline_integration.go).

func (*Engine) EnableMessageEvents 

func (e *Engine) EnableMessageEvents()

EnableMessageEvents enables RecordingStage in pipelines so message.created events are published to the event bus. This does NOT write session recordings to disk — use EnableSessionRecording for that. Requires an event bus to be configured via SetEventBus.

func (*Engine) EnableMockProviderMode 

func (e *Engine) EnableMockProviderMode(mockConfigPath string) error

EnableMockProviderMode replaces all providers in the registry with mock providers. This enables testing of scenario behavior without making real API calls.

Responses for each provider are resolved in precedence order:

  1. mockConfigPath — the --mock-config flag. An explicit operator override, so it wins for every provider.
  2. the provider's own additional_config.mock_config.
  3. the generic in-memory repository.

Step 2 is the fix for #80. This used to substitute the generic repository for every provider whenever the flag was absent, discarding each provider's declared mock_config without a warning — so a provider that was *already* a mock, with responses on disk, was replaced by one that ignored them. Every content-dependent assertion then inverted silently: on examples/guardrails-test the guardrail suite reported scenarios as passing because the replacement reply had nothing for the guardrails to catch.

Returns an error if a mock configuration file cannot be loaded or parsed.

func (*Engine) EnableSessionRecording 

func (e *Engine) EnableSessionRecording(recordingDir string) error

EnableSessionRecording enables session recording for all runs. Recordings are stored in the specified directory as JSONL files, one file per session (using RunID as session ID). Returns an error if the directory cannot be created.

func (*Engine) EnableSessionRecordingWithStore 

func (e *Engine) EnableSessionRecordingWithStore(store events.EventStore) error

EnableSessionRecordingWithStore enables session recording using a caller-provided EventStore. Programmatic Arena consumers can inject any implementation: in-memory, S3-backed, sampled, ring-buffered, etc. The dir-based EnableSessionRecording is a thin wrapper that constructs a FileEventStore.

func (*Engine) EnrichConversationTools added in v1.5.7 

func (e *Engine) EnrichConversationTools(ctx context.Context, conversationID string)

EnrichConversationTools stamps the available tools (_available_tools / _tool_descriptors) onto the conversation's system message in the state store, so the live web console's Inspector shows them — the same metadata the HTML report renders. Intended to be called after an interactive turn persists; the store's Save then re-emits the enriched system message to connected clients. No-op when the state store isn't the Arena store.

func (*Engine) EventBus added in v1.5.7 

func (e *Engine) EventBus() events.Bus

EventBus returns the engine's configured event bus (set via SetEventBus), or nil if none is configured. The web voice path wires this onto the duplex ConversationRequest so voice turns publish transcript, message, and tool events to the same bus the web SSE adapter is subscribed to — otherwise the message window stays empty during a call.

func (*Engine) ExecuteRuns 

func (e *Engine) ExecuteRuns(ctx context.Context, plan *RunPlan, concurrency int) ([]string, error)

ExecuteRuns executes all combinations in the plan and returns their run IDs. Runs are executed concurrently up to the specified concurrency limit, with run IDs collected in order matching the input plan.

Each run executes independently: - Loads scenario and provider - Executes conversation turns (with self-play if configured) - Runs validators on the results - Tracks costs, timing, and tool calls - Saves results to StateStore

The context can be used to cancel all in-flight executions. Run IDs are returned for all combinations, with errors captured in individual RunResult (accessible via StateStore).

Parameters:

  • ctx: Context for cancellation
  • plan: RunPlan containing combinations to execute
  • concurrency: Maximum number of simultaneous executions

Returns a slice of RunIDs in the same order as plan.Combinations, or an error if execution setup fails. Individual run errors are stored in StateStore, not returned here.

func (*Engine) GenerateRunPlan 

func (e *Engine) GenerateRunPlan(regionFilter, providerFilter, scenarioFilter, evalFilter []string) (*RunPlan, error)

GenerateRunPlan creates a comprehensive test execution plan from filter criteria. The plan contains all combinations of regions × providers × scenarios OR evals that match the provided filters. Scenarios and evals are mutually exclusive.

For scenarios: - regionFilter: Empty = all regions from prompt configs (or default) - providerFilter: Empty = all registered providers (or scenario-specified providers) - scenarioFilter: Empty = all loaded scenarios

For evals: - evalFilter: Empty = all loaded evals - Regions and providers are not used (they come from recordings)

Provider selection logic (scenarios only): 1. If scenario specifies providers: use those (intersected with CLI filter if provided) 2. If scenario doesn't specify providers: use all arena providers (intersected with CLI filter)

Returns a RunPlan containing all matching combinations, ready for execution. Each combination represents one independent test run that will be executed and validated separately.

func (*Engine) GetConfig 

func (e *Engine) GetConfig() *arenaconfig.Config

GetConfig returns the engine's configuration.

func (*Engine) GetDuplexExecutor 

func (e *Engine) GetDuplexExecutor() *DuplexConversationExecutor

GetDuplexExecutor returns the DuplexConversationExecutor from the engine's composite executor, or nil if the engine was not built with a duplex executor. Used by the chat command's voice path to drive RunInteractiveVoice.

func (*Engine) GetRecordingDir 

func (e *Engine) GetRecordingDir() string

GetRecordingDir returns the directory where session recordings are stored. Returns empty string if recording is not enabled.

func (*Engine) GetRecordingPath 

func (e *Engine) GetRecordingPath(runID string) string

GetRecordingPath returns the path to the recording file for a given run ID. Returns empty string if recording is not enabled.

func (*Engine) GetStateStore 

func (e *Engine) GetStateStore() statestore.Store

GetStateStore returns the engine's state store for accessing run results

func (*Engine) HasConfigEvals 

func (e *Engine) HasConfigEvals() bool

HasConfigEvals reports whether the config declares any evals the console can run for scores.

func (*Engine) ListProviders 

func (e *Engine) ListProviders() []ProviderInfo

ListProviders returns IDs of all loaded providers, sorted to put mock-style providers first (so a picker UI can default to a no-cost option) and the rest alphabetically afterwards.

func (*Engine) ListScenarios 

func (e *Engine) ListScenarios() []ScenarioInfo

ListScenarios returns IDs and descriptions of all loaded scenarios, sorted by ID.

func (*Engine) MissingRequiredVars 

func (e *Engine) MissingRequiredVars(taskType string, provided map[string]string) ([]string, error)

MissingRequiredVars returns the names of template variables the prompt for taskType declares as required but that are absent (or blank) in provided. The console uses this to prompt the user before chatting.

func (*Engine) NewInteractiveSession 

func (e *Engine) NewInteractiveSession(opts InteractiveSessionOptions) (*InteractiveSession, error)

NewInteractiveSession builds a session. Errors if the provider or scripted executor cannot be resolved. Variables are merged over the config's prompt-config vars (user wins).

func (*Engine) ProviderIDs 

func (e *Engine) ProviderIDs() []string

ProviderIDs lists the inference provider IDs declared in the loaded config, sorted for stable ordering.

func (*Engine) RegisterAudioMonitorHook 

func (e *Engine) RegisterAudioMonitorHook(hook AudioMonitorHook)

RegisterAudioMonitorHook registers a callback fired whenever the engine constructs an AudioRouter for a run. Multiple hooks can be registered; each fires in registration order. Nil hooks are ignored.

Hooks should treat the router as borrowed — Subscribe / SubscribeRMS as needed for the run's duration, but never call Close. Hook panics propagate to the goroutine running the run; treat hook bodies as application code and recover internally if needed.

func (*Engine) RegisterRunCompletedHook 

func (e *Engine) RegisterRunCompletedHook(hook RunCompletedHook)

RegisterRunCompletedHook registers a callback fired after each run in ExecuteRuns completes. Multiple hooks are supported and fire in registration order. Nil hooks are ignored. Hook panics propagate to the worker goroutine; treat hook bodies as application code.

func (*Engine) SetEventBus 

func (e *Engine) SetEventBus(bus events.Bus, opts ...EventBusOption)

SetEventBus configures the shared event bus used for runtime and TUI observability. If session recording is enabled, the event store is subscribed to the bus.

func (*Engine) SetMetrics 

func (e *Engine) SetMetrics(collector *metrics.Collector, instanceLabels map[string]string)

SetMetrics configures Prometheus metrics collection for the engine. When set, a MetricContext is created and subscribed to the event bus, recording provider call durations, token counts, costs, and tool call metrics. An event bus must be configured via SetEventBus before calling this method.

func (*Engine) SetTracerProvider 

func (e *Engine) SetTracerProvider(tp trace.TracerProvider)

SetTracerProvider configures OpenTelemetry distributed tracing for the engine. When set, an OTelEventListener is created and subscribed to the event bus, converting provider call, tool call, and pipeline events into OTel spans. An event bus must be configured via SetEventBus before calling this method.

func (*Engine) SupportsVoice added in v1.5.7 

func (e *Engine) SupportsVoice() bool

SupportsVoice reports whether the loaded config can drive a duplex voice session: any scenario declaring a Duplex block, or any provider that supports realtime audio input (see VoiceProviderIDs).

func (*Engine) VoiceProviderIDs added in v1.5.7 

func (e *Engine) VoiceProviderIDs() []string

VoiceProviderIDs lists the loaded provider IDs whose constructed provider supports realtime duplex audio input, so the web console offers the voice control only for models that can actually do it. It asks each provider its own capability (providers.StreamInputSupport.SupportsStreamInput) rather than pattern-matching config keys — so it tracks whatever the OpenAI / Gemini / mock providers report (OpenAI gates on realtime models, Gemini Live always supports it, the mock always does) and correctly excludes plain-text models. Sorted for stable ordering.

func (*Engine) WithArtifactStore 

func (e *Engine) WithArtifactStore(s artifacts.Store)

WithArtifactStore sets the backend that persists per-run report artifacts. Defaults to the local backend (bytes left in place) when unset.

func (*Engine) WithOutputDir 

func (e *Engine) WithOutputDir(dir string)

WithOutputDir sets the resolved report output directory so the engine can expose each run's artifacts base (<outputDir>/artifacts/<runID>) to hooks via SessionEvent metadata. A nil/empty dir disables artifacts-base exposure.

func (*Engine) WithSessionHooks 

func (e *Engine) WithSessionHooks(reg *hooks.Registry)

WithSessionHooks configures session lifecycle hooks on the Engine. When set, the engine fires OnSessionStart before the first turn, OnSessionUpdate after each turn, and OnSessionEnd after the conversation completes. A nil registry is a no-op — no hooks are fired.

type EvalConversationExecutor 

type EvalConversationExecutor struct {
	// contains filtered or unexported fields
}

EvalConversationExecutor handles evaluation mode: replaying saved conversations with assertions. Unlike scenario execution, eval mode: - Loads turns from recordings (no prompt building) - Applies assertions to pre-recorded assistant messages - Skips tool execution (tool calls are metadata only) - Returns results in the same schema as scenario execution for output parity

func NewEvalConversationExecutor 

func NewEvalConversationExecutor(
	adapterRegistry *adapters.Registry,
	promptRegistry *prompt.Registry,
	providerRegistry *providers.Registry,
	evalOrchestrator *EvalOrchestrator,
) *EvalConversationExecutor

NewEvalConversationExecutor creates a new eval conversation executor.

func (*EvalConversationExecutor) ExecuteConversation 

func (e *EvalConversationExecutor) ExecuteConversation(
	ctx context.Context,
	req ConversationRequest,
) *ConversationResult

ExecuteConversation runs an evaluation on a saved conversation.

func (*EvalConversationExecutor) ExecuteConversationStream 

func (e *EvalConversationExecutor) ExecuteConversationStream(
	ctx context.Context,
	req ConversationRequest,
) (<-chan ConversationStreamChunk, error)

ExecuteConversationStream runs evaluation with streaming output. For eval mode, we don't have true streaming since we're replaying, but we implement this to satisfy the interface.

type EvalOrchestrator 

type EvalOrchestrator struct {
	// contains filtered or unexported fields
}

EvalOrchestrator orchestrates eval and assertion execution during Arena runs.

func NewEvalOrchestrator 

func NewEvalOrchestrator(
	registry *evals.EvalTypeRegistry,
	defs []evals.EvalDef,
	skipEvals bool,
	evalTypeFilter []string,
	taskType string,
) *EvalOrchestrator

NewEvalOrchestrator creates a hook for executing pack evals during Arena runs. If skipEvals is true, the runner is nil and all methods are no-ops. The evalTypeFilter, when non-empty, restricts execution to matching eval types.

func (*EvalOrchestrator) Clone 

func (h *EvalOrchestrator) Clone() *EvalOrchestrator

Clone creates a shallow copy suitable for per-run use. The runner and defs are shared (immutable after construction), but metadata and workflow provider are independent. This avoids data races when concurrent runs set different workflow metadata providers.

func (*EvalOrchestrator) HasEvals 

func (h *EvalOrchestrator) HasEvals() bool

HasEvals returns true if there are eval defs to execute.

func (*EvalOrchestrator) RunAssertionsAsConversationResults 

func (h *EvalOrchestrator) RunAssertionsAsConversationResults(
	ctx context.Context,
	assertionConfigs []assertions.AssertionConfig,
	messages []types.Message,
	turnIndex int,
	sessionID string,
	trigger evals.EvalTrigger,
) []assertions.ConversationValidationResult

RunAssertionsAsConversationResults converts assertion configs to EvalDefs, runs them through the runner, and wraps results in ConversationValidationResult. The results use the original assertion type names (not pack_eval: prefixed).

func (*EvalOrchestrator) RunAssertionsAsEvals 

func (h *EvalOrchestrator) RunAssertionsAsEvals(
	ctx context.Context,
	assertionConfigs []assertions.AssertionConfig,
	messages []types.Message,
	turnIndex int,
	sessionID string,
	trigger evals.EvalTrigger,
) []evals.EvalResult

RunAssertionsAsEvals converts assertion configs to EvalDefs and runs them through the runner. Returns raw EvalResults (not converted to assertion format). The trigger parameter overrides the default trigger on each converted def.

Each assertion is converted to an EvalDef with type "assertion", which the runner dispatches to AssertionEvalHandler. The wrapper resolves the inner eval handler from the registry, executes it, and applies min_score/max_score thresholds to determine pass/fail.

func (*EvalOrchestrator) RunConversationEvals 

func (h *EvalOrchestrator) RunConversationEvals(
	ctx context.Context,
	messages []types.Message,
	sessionID string,
) []assertions.ConversationValidationResult

RunConversationEvals runs conversation-complete evals after all turns finish. Returns converted ConversationValidationResult entries.

func (*EvalOrchestrator) RunSessionEvals 

func (h *EvalOrchestrator) RunSessionEvals(
	ctx context.Context,
	messages []types.Message,
	sessionID string,
) []evals.EvalResult

RunSessionEvals runs session-complete evals after conversation finishes and returns the runtime's raw EvalResult values. Pack-level evals are observations (no pass/fail), so arena passes them through to the report unchanged — same shape that would surface in production. Use RunSessionAssertions for arena-only test assertions that do go through the gating-shaped converter.

func (*EvalOrchestrator) RunTurnEvals 

func (h *EvalOrchestrator) RunTurnEvals(
	ctx context.Context,
	messages []types.Message,
	turnIndex int,
	sessionID string,
) []assertions.ConversationValidationResult

RunTurnEvals runs turn-triggered evals after a turn completes. Returns converted ConversationValidationResult entries.

func (*EvalOrchestrator) SetClassifyRegistry 

func (h *EvalOrchestrator) SetClassifyRegistry(r *classify.Registry)

SetClassifyRegistry sets the classify.Registry that will be attached to the context passed to every eval/assertion handler. Handlers that need a classifier (audio_emotion, text_toxicity, ...) read it via classify.FromContext. Safe to call with nil — the context attach is a no-op and FromContext returns nil for handlers to surface their own error.

func (*EvalOrchestrator) SetCompositionMetadataProvider 

func (h *EvalOrchestrator) SetCompositionMetadataProvider(provider CompositionMetadataProvider)

SetCompositionMetadataProvider sets the composition observability provider for eval context injection. Called per-run for composition scenarios so composition_* assertions can access per-step outputs/branch/parallel status.

func (*EvalOrchestrator) SetEventBus 

func (h *EvalOrchestrator) SetEventBus(bus events.Bus)

SetEventBus configures the event bus for provider call telemetry in eval handlers. When set, an emitter is injected into each EvalContext's metadata so that LLM judge provider calls emit ProviderCallStarted/Completed/Failed events.

func (*EvalOrchestrator) SetMetadata 

func (h *EvalOrchestrator) SetMetadata(metadata map[string]any)

SetMetadata sets metadata that will be injected into every EvalContext. Used to pass judge_targets, prompt_registry, and other config to eval handlers.

func (*EvalOrchestrator) SetWorkflowMetadataProvider 

func (h *EvalOrchestrator) SetWorkflowMetadataProvider(provider WorkflowMetadataProvider)

SetWorkflowMetadataProvider sets the workflow state provider for eval context injection. Called per-run for workflow scenarios so assertions can access the current workflow state.

type EventBusOption 

type EventBusOption func(*eventBusConfig)

EventBusOption configures optional behavior when setting the event bus.

func WithMessageEvents 

func WithMessageEvents() EventBusOption

WithMessageEvents enables RecordingStage in pipelines so message.created events are published to the event bus. This does NOT write session recordings to disk — use EnableSessionRecording for that.

type InteractiveSession 

type InteractiveSession struct {
	// contains filtered or unexported fields
}

InteractiveSession drives one user turn at a time against a chosen agent and provider, persisting history under a fresh conversation ID. Guardrails fire automatically via the provider pipeline; assertions are never populated.

func (*InteractiveSession) ConversationID 

func (s *InteractiveSession) ConversationID() string

ConversationID returns this session's persistence key.

func (*InteractiveSession) Cost 

func (s *InteractiveSession) Cost(ctx context.Context) (types.CostInfo, error)

Cost sums per-message cost across the transcript.

func (*InteractiveSession) Messages 

func (s *InteractiveSession) Messages(ctx context.Context) ([]types.Message, error)

Messages loads the full transcript from the state store.

func (*InteractiveSession) Provider 

func (s *InteractiveSession) Provider() providers.Provider

Provider returns the resolved provider for this session. Used by the voice chat path to obtain the provider handle without a separate registry lookup.

func (*InteractiveSession) RunEvals 

func (s *InteractiveSession) RunEvals(ctx context.Context) ([]evals.EvalResult, error)

RunEvals runs the config's evals against the current transcript and returns their raw scores. Returns nil when eval scoring is disabled for the session or the config declares no evals. Evals are pure primitives (no pass/fail).

func (*InteractiveSession) RunEvalsEnabled 

func (s *InteractiveSession) RunEvalsEnabled() bool

RunEvalsEnabled reports whether eval scoring is on for this session.

func (*InteractiveSession) SendUserMessage 

func (s *InteractiveSession) SendUserMessage(
	ctx context.Context,
	text string,
) (<-chan turnexecutors.MessageStreamChunk, error)

SendUserMessage drives one live user turn and returns the assistant stream. Assertions are intentionally left nil so the assertion stage never runs; guardrails still fire inside the provider pipeline.

func (*InteractiveSession) TaskType added in v1.5.7 

func (s *InteractiveSession) TaskType() string

TaskType returns the agent (prompt config) task_type this session was created with. Voice callers (TUI and web) need it to build the arenaconfig.Scenario for a duplex run: both the ASM and composed-VAD duplex pipelines resolve the system prompt from req.Scenario.TaskType, so it must be carried over from the text session that resolved the provider.

type InteractiveSessionOptions 

type InteractiveSessionOptions struct {
	ProviderID string            // provider to chat against
	TaskType   string            // agent (prompt config) task_type
	Variables  map[string]string // user-supplied template variables
	RunEvals   bool              // when true, run config evals per turn for scores
}

InteractiveSessionOptions configures a live chat session.

type MediaOutput 

type MediaOutput struct {
	Type       string `json:"type"`                // "image", "audio", "video"
	MIMEType   string `json:"mime_type"`           // e.g., "image/jpeg", "audio/mp3"
	SizeBytes  int64  `json:"size_bytes"`          // Size of the media file
	Duration   *int   `json:"duration,omitempty"`  // Duration in seconds (for audio/video)
	Width      *int   `json:"width,omitempty"`     // Width in pixels (for image/video)
	Height     *int   `json:"height,omitempty"`    // Height in pixels (for image/video)
	FilePath   string `json:"file_path,omitempty"` // Path where media was saved
	Thumbnail  string `json:"thumbnail,omitempty"` // Base64-encoded thumbnail for HTML reports
	MessageIdx int    `json:"message_index"`       // Index of message containing this media
	PartIdx    int    `json:"part_index"`          // Index of content part within the message
}

MediaOutput represents media content generated by an LLM during test execution

func CollectMediaOutputs 

func CollectMediaOutputs(messages []types.Message) []MediaOutput

CollectMediaOutputs extracts media outputs from conversation messages Returns a slice of MediaOutput for tracking in RunResult

type MediaOutputStats 

type MediaOutputStats struct {
	Total          int            `json:"total"`
	ImageCount     int            `json:"image_count"`
	AudioCount     int            `json:"audio_count"`
	VideoCount     int            `json:"video_count"`
	TotalSizeBytes int64          `json:"total_size_bytes"`
	ByType         map[string]int `json:"by_type"`
}

MediaOutputStats contains summary statistics for media outputs

func GetMediaOutputStatistics 

func GetMediaOutputStatistics(outputs []MediaOutput) MediaOutputStats

GetMediaOutputStatistics calculates summary statistics for media outputs

type PerturbationVariant 

type PerturbationVariant struct {
	// Substitutions maps placeholder names to their values for this variant.
	Substitutions map[string]string
}

PerturbationVariant represents a single set of variable substitutions.

func ExpandPerturbations 

func ExpandPerturbations(scenario *arenaconfig.Scenario) []PerturbationVariant

ExpandPerturbations computes all perturbation variants for a scenario. It collects all perturbation maps across turns and computes the Cartesian product. Returns nil if no perturbations are defined.

type ProviderInfo 

type ProviderInfo struct {
	ID    string `json:"id"`
	Type  string `json:"type"`
	Model string `json:"model,omitempty"`
}

ProviderInfo is a lightweight view of a loaded provider for picker UIs.

type RunCombination 

type RunCombination struct {
	Region       string
	ScenarioID   string // For scenario-based runs
	EvalID       string // For eval-based runs (mutually exclusive with ScenarioID)
	ProviderID   string // Not used for eval runs (provider comes from recording)
	RecordingRef string // For batch evals: specific recording reference ID (resolved by adapter)
	TrialIndex   int    // Trial number (0-based) when scenario has Trials > 1
	TotalTrials  int    // Total number of trials for this scenario (0 or 1 = single run)

	// PerturbationIndex identifies which perturbation variant this run uses (-1 = no perturbation).
	PerturbationIndex int
}

RunCombination represents a single test execution

type RunCompletedHook 

type RunCompletedHook func(runID string, err error)

RunCompletedHook fires after each individual run finishes (success or failure). Used by the web server to persist completed runs to disk as soon as they're done, so killing the server mid-batch doesn't strand completed runs in the in-memory state store with no on-disk record.

runID is the identifier of the run that just completed. err is the run's error, or nil on success. Hooks should not retain runID across calls — the next run uses a fresh ID.

type RunPlan 

type RunPlan struct {
	Combinations []RunCombination
}

RunPlan defines the test execution plan

type RunResult 

type RunResult struct {
	RunID      string `json:"RunID"`
	PromptPack string `json:"PromptPack"`
	Region     string `json:"Region"`
	ScenarioID string `json:"ScenarioID"`
	ProviderID string `json:"ProviderID"`
	// Labels are stratification tags copied from the scenario manifest's
	// metadata.labels. Carried through to JSON/HTML reports for slicing.
	Labels     map[string]string       `json:"Labels,omitempty"`
	Params     map[string]interface{}  `json:"Params"`
	Messages   []types.Message         `json:"Messages"`
	Commit     map[string]interface{}  `json:"Commit"`
	Cost       types.CostInfo          `json:"Cost"`
	ToolStats  *types.ToolStats        `json:"ToolStats"`
	Violations []types.ValidationError `json:"Violations"`
	StartTime  time.Time               `json:"StartTime"`
	EndTime    time.Time               `json:"EndTime"`
	Duration   time.Duration           `json:"Duration"`
	Error      string                  `json:"Error"`
	Skipped    bool                    `json:"Skipped,omitempty"`
	SkipReason string                  `json:"SkipReason,omitempty"`
	SelfPlay   bool                    `json:"SelfPlay"`
	PersonaID  string                  `json:"PersonaID"`

	UserFeedback  *statestore.Feedback `json:"UserFeedback"`
	SessionTags   []string             `json:"SessionTags"`
	AssistantRole *SelfPlayRoleInfo    `json:"AssistantRole"`
	UserRole      *SelfPlayRoleInfo    `json:"UserRole"`

	// Media outputs generated by LLMs during test execution
	MediaOutputs []MediaOutput `json:"MediaOutputs,omitempty"`

	// Session recording path (if recording was enabled)
	RecordingPath string `json:"RecordingPath,omitempty"`

	// Conversation-level assertions evaluated after the conversation completes (summary format)
	ConversationAssertions AssertionsSummary `json:"conversation_assertions,omitempty"`

	// EvalResults are the runtime's pack-level eval observations
	// captured during the session — same machinery that fires in
	// production. These don't carry pass/fail; they carry structured
	// Value/Details for jq aggregation. Distinct channel from
	// ConversationAssertions so the report can render observations
	// without the gating ceremony.
	EvalResults []evals.EvalResult `json:"eval_results,omitempty"`

	// A2A agent metadata (populated from config for report rendering)
	A2AAgents []A2AAgentInfo `json:"A2AAgents,omitempty"`

	// TrialResults holds aggregated statistics when a scenario is run with Trials > 1.
	TrialResults *TrialResults `json:"trial_results,omitempty"`
}

RunResult contains the complete results of a single test execution

type ScenarioInfo 

type ScenarioInfo struct {
	ID          string `json:"id"`
	Description string `json:"description,omitempty"`
}

ScenarioInfo is a lightweight view of a loaded scenario for picker UIs.

type SelfPlayRoleInfo 

type SelfPlayRoleInfo struct {
	Provider string
	Model    string
	Region   string
}

SelfPlayRoleInfo contains provider information for self-play roles

type SkillFilterer 

type SkillFilterer interface {
	SetFilter(glob string) []string
}

SkillFilterer controls which skills are available based on workflow state.

type StateStoreConfig 

type StateStoreConfig struct {
	Store    interface{}            // State store implementation (statestore.Store)
	UserID   string                 // User identifier (optional)
	Metadata map[string]interface{} // Additional metadata to store (optional)
}

StateStoreConfig wraps the pipeline StateStore configuration for Arena

type TrialGroupKey 

type TrialGroupKey struct {
	ScenarioID string
	ProviderID string
	Region     string
}

TrialGroupKey identifies a unique scenario+provider+region combination for trial grouping.

type TrialResults 

type TrialResults struct {
	// TrialCount is the number of trials executed.
	TrialCount int `json:"trial_count"`
	// PassRate is the fraction of trials where all assertions passed (0.0-1.0).
	PassRate float64 `json:"pass_rate"`
	// FlakinessScore ranges from 0 (deterministic) to 1 (maximally flaky, 50/50).
	FlakinessScore float64 `json:"flakiness_score"`
	// PerAssertionStats maps assertion ID to its pass rate across trials.
	PerAssertionStats map[string]AssertionTrialStats `json:"per_assertion_stats,omitempty"`
}

TrialResults holds aggregated results from multiple trial executions of a scenario.

type WorkflowMetadataProvider 

type WorkflowMetadataProvider interface {
	WorkflowMetadata() map[string]any
}

WorkflowMetadataProvider is implemented by types that provide workflow state metadata for injection into the eval context during assertion evaluation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.