benchmark

Documentation

Overview

Package benchmark is the vmodel benchmark: a shared, real-world mock-provider foundation. It plays two complementary roles, and the word "benchmark" covers both — keep them distinct:

• Load generator: BenchmarkClient / BenchmarkOptions / BenchmarkResult — a pooled HTTP load driver that collects throughput / latency metrics, plus LocalServer, a thin TCP load target over virtualserver.Service.

• Reference bench: Server (bench.go) — an observable mock-provider that wraps any inner provider handler with a request-capture / hit-count layer and offers two response strategies (NewModelServer for real vmodel models, NewScenarioServer for fixture builders) over both in-process and real-TCP transports. The reusable check-logic and scenario fixtures live in the check/ and scenario/ sub-packages, consumed by protocoltest (and any other *test package or external Go project). See .design/vmodel-benchmark.md.

Both roles reuse the same vmodel registries as production code, so fixtures stay wire-format-correct. This package replaces the former pkg/benchmark, whose mock-server half duplicated virtualserver and whose Model type duplicated vmodel.Model.

Index

• Constants
• type AnthropicMessageRequest
• type BenchmarkClient
  • func NewBenchmarkClient(opts *BenchmarkOptions) *BenchmarkClient
  • func (bc *BenchmarkClient) TestChatEndpoint(model string, messages []map[string]interface{}, concurrency int, ...) (*BenchmarkResult, error)
  • func (bc *BenchmarkClient) TestMessagesEndpoint(model string, messages []map[string]interface{}, maxTokens int, ...) (*BenchmarkResult, error)
  • func (bc *BenchmarkClient) TestModelsEndpoint(concurrency int, totalRequests int) (*BenchmarkResult, error)
• type BenchmarkOptions
• type BenchmarkResult
  • func (br *BenchmarkResult) PrintSummary()
• type CapturedRequest
  • func (cr *CapturedRequest) JSON() map[string]interface{}
• type EndpointKind
• type LocalServer
  • func NewLocalServer(addr string) (*LocalServer, error)
  • func (s *LocalServer) BaseURL() string
  • func (s *LocalServer) Close() error
  • func (s *LocalServer) Port() int
  • func (s *LocalServer) Service() *virtualserver.Service
• type OpenAIChatRequest
• type RequestResult
• type Server
  • func NewModelServer() *Server
  • func NewScenarioServer() *Server
  • func NewServer(inner http.Handler) *Server
  • func (s *Server) CallCount() int
  • func (s *Server) Close() error
  • func (s *Server) EndpointHits(kind EndpointKind) int
  • func (s *Server) InProcess() string
  • func (s *Server) LastRequest(kind EndpointKind) *CapturedRequest
  • func (s *Server) LastRequestForPath(path string) *CapturedRequest
  • func (s *Server) Listen(addr string) (string, error)
  • func (s *Server) PathHits(path string) int
  • func (s *Server) Port() int
  • func (s *Server) RegisterScenario(sc scenario.Scenario)
  • func (s *Server) Reset()
  • func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (s *Server) URL() string

Constants

const DefaultPort = 12580

DefaultPort is the conventional benchmark server port. Callers may use LocalServer.Port() to discover an ephemeral port instead.

Types

type AnthropicMessageRequest

type AnthropicMessageRequest struct {
	Model     string                   `json:"model"`
	MaxTokens int                      `json:"max_tokens"`
	Messages  []map[string]interface{} `json:"messages"`
	Stream    bool                     `json:"stream,omitempty"`
}

AnthropicMessageRequest is a minimal request body for /v1/messages used by the load tester.

type BenchmarkClient

type BenchmarkClient struct {
	// contains filtered or unexported fields
}

func NewBenchmarkClient

func NewBenchmarkClient(opts *BenchmarkOptions) *BenchmarkClient

func (*BenchmarkClient) TestChatEndpoint

func (bc *BenchmarkClient) TestChatEndpoint(model string, messages []map[string]interface{}, concurrency int, totalRequests int) (*BenchmarkResult, error)

func (*BenchmarkClient) TestMessagesEndpoint

func (bc *BenchmarkClient) TestMessagesEndpoint(model string, messages []map[string]interface{}, maxTokens int, concurrency int, totalRequests int) (*BenchmarkResult, error)

func (*BenchmarkClient) TestModelsEndpoint

func (bc *BenchmarkClient) TestModelsEndpoint(concurrency int, totalRequests int) (*BenchmarkResult, error)

type BenchmarkOptions

type BenchmarkOptions struct {
	BaseURL          string
	Timeout          time.Duration
	MaxConns         int
	Provider         string // "openai" or "anthropic"
	APIKey           string
	DisableKeepAlive bool
}

type BenchmarkResult

type BenchmarkResult struct {
	TotalRequests    int           `json:"totalRequests"`
	SuccessRequests  int           `json:"successRequests"`
	FailedRequests   int           `json:"failedRequests"`
	TotalDuration    time.Duration `json:"totalDuration"`
	AvgResponseTime  time.Duration `json:"avgResponseTime"`
	MinResponseTime  time.Duration `json:"minResponseTime"`
	MaxResponseTime  time.Duration `json:"maxResponseTime"`
	RequestsPerSec   float64       `json:"requestsPerSec"`
	TotalBytes       int64         `json:"totalBytes"`
	ErrorRate        float64       `json:"errorRate"`
	StatusCodeCounts map[int]int   `json:"statusCodeCounts"`
}

func (*BenchmarkResult) PrintSummary

func (br *BenchmarkResult) PrintSummary()

type CapturedRequest

type CapturedRequest struct {
	Method  string
	Path    string
	Headers http.Header
	Body    []byte
}

CapturedRequest records what the gateway forwarded to a provider endpoint so observers can assert on the outbound request (model, flags, headers, body).

func (*CapturedRequest) JSON

func (cr *CapturedRequest) JSON() map[string]interface{}

JSON decodes the captured body into a generic map. Returns an empty map if the body is absent or not JSON.

type EndpointKind

type EndpointKind string

EndpointKind identifies which provider-native endpoint a request hit. chat and responses are deliberately distinct: they are two different OpenAI protocols, and observers assert which one was actually forwarded to.

This is a test-observability axis ("which route did this request hit"), cross-provider, distinct from two adjacent production enums it must NOT be conflated with: ai.OpenAIEndpointMode ({unknown,chat,responses,both}) is a provider *configuration* (which OpenAI endpoints a provider supports, OpenAI only), and ai.APIType / APIStyle is the protocol *family*. EndpointKind only shares the "chat"/"responses" literals with the former — different concept, different layer.

const (
	EndpointChat      EndpointKind = "chat"
	EndpointResponses EndpointKind = "responses"
	EndpointAnthropic EndpointKind = "anthropic"
	EndpointGoogle    EndpointKind = "google"
	EndpointUnknown   EndpointKind = "unknown"
)

type LocalServer

type LocalServer struct {
	// contains filtered or unexported fields
}

LocalServer is the capture-free load target: a virtualserver.Service exposed over a real HTTP listener so the benchmark load client can hit a loopback target with no per-request observability overhead. It shares route wiring with the observable reference Server via modelRouter (see bench.go) but deliberately omits the capture middleware — for an observable server (request capture, endpoint-hit counts) use NewModelServer().Listen() instead.

func NewLocalServer

func NewLocalServer(addr string) (*LocalServer, error)

NewLocalServer starts an in-process benchmark server bound to addr (an empty string or ":0" picks an ephemeral port). The returned server is already listening; call Port() to discover the bound port and Close() to shut down. The underlying virtualmodel registries come pre-populated with the same defaults as production via virtualserver.NewService.

func (*LocalServer) BaseURL

func (s *LocalServer) BaseURL() string

BaseURL returns http://localhost:<port> for use as BenchmarkOptions.BaseURL.

func (*LocalServer) Close

func (s *LocalServer) Close() error

Close shuts down the server with a short grace period.

func (*LocalServer) Port

func (s *LocalServer) Port() int

Port returns the TCP port the server is listening on.

func (*LocalServer) Service

func (s *LocalServer) Service() *virtualserver.Service

Service returns the underlying virtualserver.Service so callers can register additional virtual models on its anthropic / openai registries.

type OpenAIChatRequest

type OpenAIChatRequest struct {
	Model    string                   `json:"model"`
	Messages []map[string]interface{} `json:"messages"`
	Stream   bool                     `json:"stream,omitempty"`
}

OpenAIChatRequest is a minimal request body for /v1/chat/completions used by the load tester. It is not a full SDK type — only the fields the benchmark client needs to construct a request.

type RequestResult

type RequestResult struct {
	Duration   time.Duration
	StatusCode int
	Error      error
	Bytes      int64
}

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is the observable reference mock-provider at the heart of the vmodel benchmark. It wraps any inner provider http.Handler with a capture middleware that records request counts, per-endpoint hits, and the last forwarded request — observability that is independent of how responses are generated.

Response generation is pluggable via the constructor used:

• NewModelServer() serves real vmodel models (protocol-correct bytes), the same registries that back the production /virtual/v1/* endpoint.
• NewScenarioServer() serves registered scenario fixtures across all four provider formats (OpenAI chat / responses, Anthropic, Google).
• NewServer(inner) wraps an arbitrary provider handler.

Transport is also pluggable on the same instance: InProcess() starts an httptest server (in-process), Listen() binds a real TCP port (for subprocess or external clients). The type carries no *testing.T dependency, so it can be imported by external Go projects as well as test packages.

func NewModelServer

func NewModelServer() *Server

NewModelServer builds a Server whose inner handler is the production virtualserver.Service mounted under /v1, /openai/v1, and /anthropic/v1 — the same wiring (and default model registries) as the production endpoint, so the responses are wire-format-correct. Use this for servertest, load tests, and external projects that want a realistic provider.

func NewScenarioServer

func NewScenarioServer() *Server

NewScenarioServer builds a Server whose inner handler serves registered scenario fixtures (scenario.MockResponseBuilder) across all four provider formats. Register scenarios with RegisterScenario. Use this for the protocol transform matrix and for byte-exact mocks where you control the exact response.

func NewServer

func NewServer(inner http.Handler) *Server

NewServer wraps an arbitrary provider handler with the capture middleware.

func (*Server) CallCount

func (s *Server) CallCount() int

CallCount returns the total number of requests received.

func (*Server) Close

func (s *Server) Close() error

Close shuts the server down with a short grace period.

func (*Server) EndpointHits

func (s *Server) EndpointHits(kind EndpointKind) int

EndpointHits returns how many requests hit a specific provider endpoint.

func (*Server) InProcess

func (s *Server) InProcess() string

InProcess starts the server on an in-process httptest listener and returns its base URL. Prefer this for in-process Go tests.

func (*Server) LastRequest

func (s *Server) LastRequest(kind EndpointKind) *CapturedRequest

LastRequest returns the most recent request forwarded to the given provider endpoint, or nil if that endpoint was never hit.

func (*Server) LastRequestForPath

func (s *Server) LastRequestForPath(path string) *CapturedRequest

LastRequestForPath returns the most recent request captured for one exact URL path, or nil if that path was never hit.

func (*Server) Listen

func (s *Server) Listen(addr string) (string, error)

Listen binds a real TCP listener (addr "" or ":0" picks an ephemeral port) and returns the base URL. Prefer this when a subprocess or external client must reach the server over loopback.

func (*Server) PathHits

func (s *Server) PathHits(path string) int

PathHits returns how many requests were captured for one exact URL path.

func (*Server) Port

func (s *Server) Port() int

Port returns the TCP port for a Listen()-started server, or 0 otherwise.

func (*Server) RegisterScenario

func (s *Server) RegisterScenario(sc scenario.Scenario)

RegisterScenario registers (or replaces) a scenario on a scenario server. It is a no-op on a model server. The registry ordinarily errors on duplicate IDs, so a prior entry with the same name is cleared first.

func (*Server) Reset

func (s *Server) Reset()

Reset clears all recorded counts and captured requests.

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP lets a Server be used directly as an http.Handler while preserving the same capture and endpoint accounting as its managed transports.

func (*Server) URL

func (s *Server) URL() string

URL returns the base URL the server is reachable at (empty until InProcess or Listen has been called).