protocol_validate

package
v0.260430.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 26, 2026 License: MPL-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package protocoltest provides a framework for end-to-end validation of the model gateway's protocol transformation layer.

Architecture

  1. server_validate.VirtualServer — a mock HTTP provider that speaks OpenAI, Anthropic, and Google response formats. Conceptually a "virtual model" for testing.

  2. TestEnv — wires a real gateway Server (with transform pipeline) to a VirtualServer, configures routing rules, and provides SendAs() for round-trip testing.

  3. Matrix — executes the full cross-product of sources × targets × scenarios × streaming modes.

Note: The existing internal/virtualmodel package is a production Gin server. This package (protocoltest) is the test-only framework. Future integration with virtualmodel is planned once both stabilize.

Usage 

env := protocoltest.NewTestEnv(t)
defer env.Close()
env.SetupRoute(protocol.TypeAnthropicV1, protocol.TypeOpenAIChat, protocoltest.TextScenario())
result := env.SendAs(t, protocol.TypeAnthropicV1, protocoltest.TextScenario(), false)
assert.Equal(t, "assistant", result.Role)

Index

Constants

View Source 
const (
	ServerModeAuto = "auto" // Per-scenario (default)
	ServerModeAll  = "all"  // Single server for all tests
	ServerModePair = "pair" // Per source-target pair
)

ServerMode constants

Variables

This section is empty.

Functions

func ResolveAPIStyle added in v0.260423.0 

func ResolveAPIStyle(entry RealModelEntry) (string, error)

ResolveAPIStyle returns the effective api_style for an entry. Returns an error if api_style is empty or contains an invalid value. Valid values are: "openai", "anthropic", "google".

func ResolveAPIType added in v0.260423.0 

func ResolveAPIType(entry RealModelEntry) (string, error)

ResolveAPIType returns the effective api_type for an entry. If the entry specifies one, it is validated and returned. If empty, returns a default based on api_style:

  • "anthropic" → "anthropic_v1"
  • "openai" → "openai_chat"
  • "google" → "google"

Types

type AgentScenario added in v0.260423.0 

type AgentScenario struct {
	// Name is the scenario name
	Name string

	// Description describes what the scenario tests
	Description string

	// MockResponses are the mock responses keyed by API style
	MockResponses map[server_validate.ResponseFormat]server_validate.MockResponseBuilder
}

AgentScenario defines a test scenario for profile testing Unlike matrix scenarios which test protocol transformations, profile scenarios test client implementations (OAuth, path rewriting, etc.)

func AgentScenarios added in v0.260423.0 

func AgentScenarios() []AgentScenario

AgentScenarios returns all built-in profile test scenarios

func AgentStreamingTextScenario added in v0.260423.0 

func AgentStreamingTextScenario() AgentScenario

AgentStreamingTextScenario returns a scenario for streaming text response

func AgentTextScenario added in v0.260423.0 

func AgentTextScenario() AgentScenario

AgentTextScenario returns a scenario for basic text request/response

func AgentToolUseScenario added in v0.260423.0 

func AgentToolUseScenario() AgentScenario

AgentToolUseScenario returns a scenario for tool use request/response

func GetScenarioByName 

func GetScenarioByName(name string) (AgentScenario, bool)

GetScenarioByName returns a profile scenario by name

type AgentTestEnv added in v0.260423.0 

type AgentTestEnv struct {
	// contains filtered or unexported fields
}

AgentTestEnv provides an isolated test environment for Agent testing It includes: - A temporary config directory - A gateway server with virtual provider - Routing rules configured for the Agent - A virtual server that captures requests for validation

func NewAgentTestEnv added in v0.260423.0 

func NewAgentTestEnv(AgentType AgentType) (*AgentTestEnv, error)

NewAgentTestEnv creates a new Agent test environment The environment is isolated with a temporary config directory and must be cleaned up with Close() when done

func (*AgentTestEnv) AppConfig added in v0.260423.0 

func (env *AgentTestEnv) AppConfig() *serverconfig.Config

AppConfig returns the application configuration

func (*AgentTestEnv) BaseURL added in v0.260423.0 

func (env *AgentTestEnv) BaseURL() string

BaseURL returns the base URL of the gateway server

func (*AgentTestEnv) Close added in v0.260423.0 

func (env *AgentTestEnv) Close(preserve bool) error

Close cleans up the test environment If preserve is true, the config directory is kept for inspection

func (*AgentTestEnv) ConfigDir added in v0.260423.0 

func (env *AgentTestEnv) ConfigDir() string

ConfigDir returns the temporary config directory path

func (*AgentTestEnv) ModelToken added in v0.260423.0 

func (env *AgentTestEnv) ModelToken() string

ModelToken returns the model token for requests

func (*AgentTestEnv) SetupAgent added in v0.260423.0 

func (env *AgentTestEnv) SetupAgent(AgentType AgentType, providerName string, modelName string) error

SetupAgent configures the environment for a specific Agent type This creates the necessary provider and routing rules

func (*AgentTestEnv) SetupRealAgent added in v0.260423.0 

func (env *AgentTestEnv) SetupRealAgent(AgentType AgentType, providerName string, modelName string, apiBase string, apiKey string, apiStyle string) error

SetupRealAgent configures the environment to route through a real upstream provider. Unlike SetupAgent, it does not use the virtual server — the provider points at the real apiBase with the real apiKey. apiStyle must be "openai" or "anthropic". apiType is optional and specifies the target API type (e.g., "anthropic_v1", "openai_chat"). If empty, a default is chosen based on apiStyle.

func (*AgentTestEnv) VirtualServerURL added in v0.260423.0 

func (env *AgentTestEnv) VirtualServerURL() string

VirtualServerURL returns the URL of the virtual server

type AgentTestResult added in v0.260423.0 

type AgentTestResult struct {
	// Name is the test name
	Name string

	// Agent is the Agent type being tested
	Agent AgentType

	// Scenario is the test scenario (e.g., "text", "streaming", "tool_use")
	Scenario string

	// Passed indicates whether the test passed
	Passed bool

	// Skipped indicates whether the test was skipped
	Skipped bool

	// SkipReason explains why the test was skipped
	SkipReason string

	// Errors contains any assertion errors
	Errors []AssertionError

	// Duration is how long the test took
	Duration int64 // milliseconds

	// HTTPStatus is the HTTP status code received
	HTTPStatus int

	// RequestHeaders contains the request headers sent to the virtual server
	RequestHeaders http.Header

	// RequestBody contains the request body sent to the virtual server
	RequestBody []byte

	// ResponseBody contains the raw response body
	ResponseBody []byte
}

AgentTestResult represents the result of a single Agent test

type AgentType added in v0.260423.0 

type AgentType string

AgentType represents the type of agent Agent to test 

const (
	AgentTypeClaudeCode AgentType = "claude"
	AgentTypeCodex      AgentType = "codex"
	AgentTypeOpenCode   AgentType = "opencode"
)

func (AgentType) Scenario added in v0.260423.0 

func (pt AgentType) Scenario() typ.RuleScenario

Scenario returns the corresponding RuleScenario for this Agent

func (AgentType) String added in v0.260423.0 

func (pt AgentType) String() string

String returns the string representation of AgentType

type Assertion 

type Assertion struct {
	Name  string
	Check func(r *RoundTripResult) error
}

Assertion is a named check applied to a RoundTripResult.

func AssertContentContains 

func AssertContentContains(substring string) Assertion

AssertContentContains returns an Assertion that the response content contains substring.

func AssertContentEquals 

func AssertContentEquals(expected string) Assertion

AssertContentEquals returns an Assertion that the response content equals expected.

func AssertFinishReason 

func AssertFinishReason(expected string) Assertion

AssertFinishReason returns an Assertion that the finish_reason equals expected.

func AssertHTTPStatus 

func AssertHTTPStatus(expected int) Assertion

AssertHTTPStatus returns an Assertion that the HTTP status code equals expected.

func AssertHasThinking 

func AssertHasThinking() Assertion

AssertHasThinking returns an Assertion that thinking content is non-empty.

func AssertHasToolCalls 

func AssertHasToolCalls(count int) Assertion

AssertHasToolCalls returns an Assertion that exactly count tool calls are present.

func AssertModelContains 

func AssertModelContains(substring string) Assertion

AssertModelContains returns an Assertion that the model name contains substring.

func AssertNoThinking 

func AssertNoThinking() Assertion

AssertNoThinking returns an Assertion that no thinking content is present.

func AssertRoleEquals 

func AssertRoleEquals(expected string) Assertion

AssertRoleEquals returns an Assertion that the response role equals expected.

func AssertStreamEventCount 

func AssertStreamEventCount(min int) Assertion

AssertStreamEventCount returns an Assertion that at least min SSE events were received.

func AssertToolCallArgs 

func AssertToolCallArgs(index int, key, value string) Assertion

AssertToolCallArgs returns an Assertion that tool call at index has key=value in its JSON args.

func AssertToolCallName 

func AssertToolCallName(index int, name string) Assertion

AssertToolCallName returns an Assertion that the tool call at index has the given name.

func AssertUsageNonZero 

func AssertUsageNonZero() Assertion

AssertUsageNonZero returns an Assertion that at least one token count > 0.

type AssertionError 

type AssertionError struct {
	Assertion string // assertion name
	Error     string // error message
	Context   string // additional context (truncated body, etc.)
}

AssertionError represents a single assertion failure.

type CapturedRequest 

type CapturedRequest struct {
	// Headers contains the request headers
	Headers http.Header

	// Body contains the request body
	Body []byte

	// Method is the HTTP method
	Method string

	// Path is the request path
	Path string
}

CapturedRequest represents a request captured by the virtual server

type Matrix 

type Matrix struct {
	Sources    []protocol.APIType
	Targets    []protocol.APIType
	Scenarios  []Scenario
	Streaming  []bool
	RecordDir  string // Optional directory for recording requests/responses
	ServerMode string // Server reuse mode: auto, all, pair
	BatchCount int    // Number of times to run each test
}

Matrix defines the cross-product of source protocols, target protocols, scenarios, and streaming modes to validate.

func DefaultMatrix 

func DefaultMatrix() *Matrix

DefaultMatrix returns the full validation matrix covering all supported protocol combinations, all built-in scenarios, and both streaming modes.

func (*Matrix) ExecuteAll 

func (m *Matrix) ExecuteAll() []TestResult

ExecuteAll runs all matrix combinations and returns structured results. This is a pure function that can be called from both tests and CLI. It does not use testing.T, making it suitable for standalone execution.

Server reuse strategies based on ServerMode: - auto (default): Reuses TestEnv per scenario - all: Uses a single TestEnv for all tests (fastest, potential for interference) - pair: Reuses TestEnv per source-target pair (balanced approach)

func (*Matrix) OnlyScenarios 

func (m *Matrix) OnlyScenarios(names ...string) *Matrix

OnlyScenarios returns a copy of the Matrix filtered to only the named scenarios.

func (*Matrix) OnlySources added in v0.260430.1 

func (m *Matrix) OnlySources(sources ...string) *Matrix

OnlySources returns a copy of the Matrix filtered to only the specified source protocols.

func (*Matrix) OnlyStreaming added in v0.260430.1 

func (m *Matrix) OnlyStreaming(streaming bool) *Matrix

OnlyStreaming returns a copy of the Matrix filtered to only streaming or non-streaming tests. If streaming is true, only streaming tests are included. If false, only non-streaming tests.

func (*Matrix) OnlyTargets added in v0.260430.1 

func (m *Matrix) OnlyTargets(targets ...string) *Matrix

OnlyTargets returns a copy of the Matrix filtered to only the specified target protocols.

func (*Matrix) Run 

func (m *Matrix) Run(t *testing.T)

Run executes all matrix combinations as subtests under t. Each combination runs in its own TestEnv so state is isolated.

func (*Matrix) WithBatchCount added in v0.260430.1 

func (m *Matrix) WithBatchCount(count int) *Matrix

WithBatchCount returns a copy of the Matrix with the batch count set.

func (*Matrix) WithRecordDir added in v0.260430.1 

func (m *Matrix) WithRecordDir(recordDir string) *Matrix

WithRecordDir returns a copy of the Matrix with the record directory set. If recordDir is empty, recording is disabled.

func (*Matrix) WithServerMode added in v0.260430.1 

func (m *Matrix) WithServerMode(mode string) *Matrix

WithServerMode returns a copy of the Matrix with the server reuse mode set.

type MockResponseBuilder 

type MockResponseBuilder = server_validate.MockResponseBuilder

MockResponseBuilder re-exports server_validate.MockResponseBuilder for convenience.

type ProviderConfig added in v0.260423.0 

type ProviderConfig struct {
	Name     string   `yaml:"name"`
	BaseURL  string   `yaml:"baseurl"`
	APIKey   string   `yaml:"apikey"`
	APIStyle string   `yaml:"api_style"` // required: "openai" | "anthropic" | "google"
	APIType  string   `yaml:"api_type"`  // optional: "openai_chat" | "openai_responses" | "anthropic_v1" | "anthropic_beta" | "google"
	Models   []string `yaml:"models"`    // list of model names to test
}

ProviderConfig is one provider entry in the config YAML file. A provider can have multiple models under it.

type ProvidersConfig added in v0.260423.0 

type ProvidersConfig struct {
	Providers []ProviderConfig `yaml:"providers"`
}

ProvidersConfig is the top-level structure of the config YAML file.

type RealModelEntry added in v0.260423.0 

type RealModelEntry struct {
	Name     string // generated entry name: "provider" or "provider-model"
	Provider string // original provider name
	BaseURL  string
	APIKey   string
	Model    string
	APIStyle string
	APIType  string
}

RealModelEntry is an expanded entry for testing. Each (provider, model) pair becomes one entry.

func ExpandProvidersConfig added in v0.260423.0 

func ExpandProvidersConfig(cfg *ProvidersConfig) []RealModelEntry

ExpandProvidersConfig expands a ProvidersConfig into individual test entries. Each provider's models array is expanded into separate entries.

func LoadProvidersConfig added in v0.260423.0 

func LoadProvidersConfig(path string) ([]RealModelEntry, error)

LoadProvidersConfig reads and parses a providers config YAML file. Returns the expanded list of test entries.

type RealModelsConfig added in v0.260423.0 

type RealModelsConfig struct {
	Models []RealModelEntry
}

RealModelsConfig is the legacy format kept for backward compatibility. Deprecated: Use ProvidersConfig instead.

func LoadRealModelsConfig added in v0.260423.0 

func LoadRealModelsConfig(path string) (*RealModelsConfig, error)

LoadRealModelsConfig is an alias for LoadProvidersConfig for backward compatibility. Deprecated: Use LoadProvidersConfig instead.

type RoundTripResult 

type RoundTripResult struct {
	// Source and target context
	SourceProtocol protocol.APIType
	TargetProtocol protocol.APIType
	ScenarioName   string
	IsStreaming    bool

	// HTTP layer
	HTTPStatus   int
	RawBody      []byte
	StreamEvents []string // raw SSE event lines (streaming only)

	// Extracted semantics (populated by the framework after parsing)
	Content         string
	Role            string
	Model           string
	FinishReason    string
	ToolCalls       []ToolCallResult
	ThinkingContent string
	Usage           *TokenUsage
}

RoundTripResult captures the full result of a round-trip request through the gateway.

type Scenario 

type Scenario struct {
	Name        string
	Description string
	Tags        []string

	// MockResponses keyed by response format (openai_chat, openai_responses, anthropic, google).
	MockResponses map[server_validate.ResponseFormat]MockResponseBuilder

	// Assertions run after every round-trip for this scenario.
	Assertions []Assertion
}

Scenario is a named test scenario describing:

  • What the mock provider should return (MockResponses per ResponseFormat)
  • What assertions to run on the round-trip result

func AllScenarios 

func AllScenarios() []Scenario

AllScenarios returns the full set of built-in validation scenarios.

func ErrorScenario 

func ErrorScenario() Scenario

ErrorScenario tests that provider error responses are forwarded to the client.

func MultiTurnScenario 

func MultiTurnScenario() Scenario

MultiTurnScenario tests a conversation with system prompt + 2 turns of history.

func StreamingTextScenario 

func StreamingTextScenario() Scenario

StreamingTextScenario tests SSE streaming for a plain text response.

func StreamingToolUseScenario 

func StreamingToolUseScenario() Scenario

StreamingToolUseScenario tests SSE streaming for a tool call response.

func TextScenario 

func TextScenario() Scenario

TextScenario is the baseline: a single user message and a plain text reply.

func ThinkingScenario 

func ThinkingScenario() Scenario

ThinkingScenario tests that extended thinking blocks are present in Anthropic responses.

func ToolResultScenario 

func ToolResultScenario() Scenario

ToolResultScenario tests a multi-turn conversation including a tool result message.

func ToolUseScenario 

func ToolUseScenario() Scenario

ToolUseScenario exercises a single tool/function call.

type TestEnv 

type TestEnv struct {
	// contains filtered or unexported fields
}

TestEnv wires a real gateway Server (with the full transform pipeline) to a VirtualServer (mock provider). It manages config, routing rules, and provides SendAs() for full round-trip testing.

**Routing Architecture**: 

Client Request → Gateway (/tingly/{scenario}/v1/...)
              → Protocol Transform
              → Provider Request (virtual-server-url/v1/...)
              → VirtualServer (mock provider response)

The gateway handles /tingly/{scenario}/v1/... routes, transforms the request to provider format, and forwards to the virtual server which speaks provider native APIs (/v1/chat/completions, /v1/messages, etc.).

func NewTestEnv 

func NewTestEnv(t *testing.T) *TestEnv

NewTestEnv creates a TestEnv with a fresh gateway config and a new VirtualServer. All resources are cleaned up via t.Cleanup.

func NewTestEnvForCLI 

func NewTestEnvForCLI(opts ...TestEnvOption) (*TestEnv, error)

NewTestEnvForCLI creates a TestEnv for CLI use (without testing.T). Resources must be cleaned up via explicit Close() call.

func (*TestEnv) Close 

func (env *TestEnv) Close()

Close cleans up resources. For testing mode, it's a no-op (resources are cleaned up via t.Cleanup). For CLI mode, it closes the servers and removes the config directory.

func (*TestEnv) SendAs 

func (env *TestEnv) SendAs(t *testing.T, source, target protocol.APIType, s Scenario, streaming bool) *RoundTripResult

SendAs sends a request to the gateway as the given source protocol, using the request model configured by SetupRoute, and returns the parsed result.

Streaming requests use the real httptest.Server (env.gatewayServer) because httptest.ResponseRecorder does not support Gin's streaming/SSE machinery. Non-streaming requests use the recorder for simplicity.

func (*TestEnv) SendAsCLI 

func (env *TestEnv) SendAsCLI(source, target protocol.APIType, s Scenario, streaming bool) (*RoundTripResult, error)

SendAsCLI sends a request to the gateway as the given source protocol, using the request model configured by SetupRoute, and returns the parsed result. This version is for CLI use and returns errors instead of calling t.Fatalf.

Streaming requests use the real httptest.Server (env.gatewayServer) because httptest.ResponseRecorder does not support Gin's streaming/SSE machinery. Non-streaming requests use the recorder for simplicity.

func (*TestEnv) SetupRoute 

func (env *TestEnv) SetupRoute(source, target protocol.APIType, s Scenario)

SetupRoute configures a gateway rule that routes source protocol requests to the virtual server acting as a target protocol provider.

The virtual server is pre-registered with the scenario's mock responses. If the route has already been set up, this is a no-op (idempotent).

**Routing Flow**: 1. Client sends request to gateway: POST /tingly/{scenario}/v1/chat/completions 2. Gateway transforms request to provider format based on source protocol 3. Gateway forwards to provider: POST {virtualURL}/v1/chat/completions 4. VirtualServer (provider mock) returns pre-configured scenario response

The provider's APIBase includes the /v1 suffix for OpenAI-style providers to match actual provider API structure.

func (*TestEnv) VirtualCallCount 

func (env *TestEnv) VirtualCallCount() int

VirtualCallCount returns the number of requests received by the virtual server.

func (*TestEnv) VirtualURL 

func (env *TestEnv) VirtualURL() string

VirtualURL returns the URL of the underlying virtual server.

type TestEnvOption added in v0.260430.1 

type TestEnvOption func(*testEnvConfig)

TestEnvOption is a functional option for configuring TestEnv.

func NewTestEnvOptionWithRecordDir added in v0.260430.1 

func NewTestEnvOptionWithRecordDir(dir string) TestEnvOption

NewTestEnvOptionWithRecordDir creates an option to set the record directory. If empty, recording is disabled.

type TestResult 

type TestResult struct {
	// Test identification
	Name      string // Full test name: "scenario/source/target/mode"
	Scenario  string // Scenario name: "text", "tool_use", etc.
	Source    protocol.APIType
	Target    protocol.APIType
	Streaming bool

	// Test outcome
	Passed     bool   // true if all assertions passed
	Skipped    bool   // true if test was skipped
	SkipReason string // reason for skipping

	// Error details
	Errors   []AssertionError // list of assertion failures
	Duration time.Duration    // test execution time

	// Batch statistics (populated when BatchCount > 1)
	BatchCount  int           // number of times the test was executed
	BatchPassed int           // number of executions that passed
	BatchMinDur time.Duration // minimum duration across executions
	BatchAvgDur time.Duration // average duration across executions
	BatchMaxDur time.Duration // maximum duration across executions
	BatchErrors []string      // unique error messages from failed executions

	// Response details (for debugging/verbose output)
	HTTPStatus int              // HTTP status code
	Response   *RoundTripResult // full round-trip result (from first or last execution)
}

TestResult represents the outcome of a single matrix test combination. This is returned by Matrix.ExecuteAll() for CLI and other non-testing contexts.

type TokenUsage 

type TokenUsage struct {
	InputTokens  int
	OutputTokens int
}

TokenUsage holds token counts extracted from a provider response.

type ToolCallResult 

type ToolCallResult struct {
	ID        string
	Name      string
	Arguments string // raw JSON string
}

ToolCallResult holds a single tool/function call extracted from a response.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.