benchmark

package
v0.260224.1130 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 24, 2026 License: MPL-2.0 Imports: 9 Imported by: 0

README

Benchmark Package

A comprehensive benchmark package for testing LLM API proxies with support for OpenAI and Anthropic compatible endpoints.

Features

  • Mock Server: Configurable mock server that simulates OpenAI and Anthropic API endpoints
  • Benchmark Client: High-performance client for load testing services
  • Flexible Configuration: JSON-based configuration for custom responses, delays, and behavior
  • Multiple Providers: Support for both OpenAI and Anthropic API formats
  • Gin Framework: Built on Gin HTTP framework for better performance and extensibility
  • Detailed Metrics: Comprehensive performance metrics including response times, throughput, and error rates

Quick Start

1. Create a Configuration
# Create a sample configuration file
go run pkg/benchmark/examples/server/main.go config.json
2. Start the Mock Server
# Edit the config.json file as needed, then run:
go run pkg/benchmark/examples/server/main.go config.json
3. Run Benchmarks
# Run benchmark tests
go run pkg/benchmark/examples/client/main.go

Configuration

The mock server uses a JSON configuration file to define its behavior:

{
  "port": 8080,
  "provider": "openai",
  "models": {
    "defaultList": [
      {
        "id": "gpt-3.5-turbo",
        "object": "model",
        "created": 1677610602,
        "owned_by": "openai"
      }
    ]
  },
  "chat": {
    "defaultResponses": [
      {
        "id": "chatcmpl-123",
        "object": "chat.completion",
        "created": 1677652288,
        "model": "gpt-3.5-turbo",
        "choices": [
          {
            "index": 0,
            "message": {
              "role": "assistant",
              "content": "Hello! This is a mock response."
            },
            "finish_reason": "stop"
          }
        ],
        "usage": {
          "prompt_tokens": 10,
          "completion_tokens": 9,
          "total_tokens": 19
        }
      }
    ],
    "loopResponses": true,
    "delayMs": 100
  },
  "message": {
    "defaultResponses": [...],
    "loopResponses": true,
    "delayMs": 150
  }
}
Configuration Options
  • port: Server port (default: 8080)
  • provider: API provider type ("openai" or "anthropic")
  • models.defaultList: List of models to return from /v1/models
  • chat.defaultResponses: Array of chat completion responses to cycle through
  • chat.loopResponses: Whether to loop through responses (default: true)
  • chat.delayMs: Artificial delay in milliseconds for chat responses
  • message: Similar configuration for Anthropic message endpoints

Architecture

The mock server is built using the Gin HTTP framework, providing:

  • High performance and low memory footprint
  • Flexible routing with middleware support
  • Built-in CORS support for cross-origin requests
  • Easy extensibility for new endpoints and features

API Endpoints

OpenAI Compatible Endpoints
  • GET /v1/models - List available models
  • POST /v1/chat/completions - Create chat completion
  • GET /openai/v1/models - OpenAI-specific models endpoint
  • POST /openai/v1/chat/completions - OpenAI-specific chat endpoint
Anthropic Compatible Endpoints
  • POST /v1/messages - Create message
  • POST /anthropic/v1/messages - Anthropic-specific message endpoint

Usage Examples

Using the Mock Server Directly
package main

import (
    "log"
    "your-project/pkg/benchmark"
)

func main() {
    // Create a mock server with custom options
    server := benchmark.NewMockServer(
        benchmark.WithPort(8080),
        benchmark.WithOpenAIDefaults(),
        benchmark.WithChatDelay(100),
    )

    // Start the server
    log.Printf("Starting mock server on port %d", server.Port())
    if err := server.Start(); err != nil {
        log.Fatal(err)
    }
}
Using the Benchmark Client
package main

import (
    "fmt"
    "time"
    "your-project/pkg/benchmark"
)

func main() {
    // Create client
    client := benchmark.NewBenchmarkClient(&benchmark.BenchmarkOptions{
        BaseURL:  "http://localhost:8080",
        Provider: "openai",
        APIKey:   "sk-test-key",
        Timeout:  30 * time.Second,
        MaxConns: 100,
    })

    // Test models endpoint
    result, err := client.TestModelsEndpoint(10, 100)
    if err != nil {
        panic(err)
    }
    result.PrintSummary()

    // Test chat completions
    messages := []benchmark.ChatMessage{
        {Role: "user", Content: "Hello, how are you?"},
    }
    result, err = client.TestChatEndpoint("gpt-3.5-turbo", messages, 20, 200)
    if err != nil {
        panic(err)
    }
    result.PrintSummary()
}
Running Tests
# Run all tests
go test ./pkg/benchmark/...

# Run with verbose output
go test -v ./pkg/benchmark/...

# Run benchmarks
go test -bench=. ./pkg/benchmark/...

Metrics

The benchmark client provides comprehensive performance metrics:

  • Total Requests: Number of requests sent
  • Success/Failed Requests: Success and failure counts
  • Response Times: Average, minimum, and maximum response times
  • Throughput: Requests per second
  • Error Rate: Percentage of failed requests
  • Status Code Distribution: Breakdown by HTTP status codes
  • Total Bytes: Total amount of data transferred

Contributing

When adding new features:

  1. Ensure backward compatibility
  2. Add appropriate tests
  3. Update documentation
  4. Follow Go conventions and best practices

License

This package is part of the tingly-box project.

Documentation

Index

Examples

Constants

View Source 
const (
	DefaultPort           = 12580
	DefaultChatDelayMs    = 100
	DefaultMessageDelayMs = 150
	DefaultRandomDelayMin = 10
	DefaultRandomDelayMax = 100
)

Default configuration constants

Variables

This section is empty.

Functions

This section is empty.

Types

type AnthropicMessageRequest 

type AnthropicMessageRequest struct {
	Model     string                   `json:"model"`
	MaxTokens int                      `json:"max_tokens"`
	Messages  []map[string]interface{} `json:"messages"`
	Stream    bool                     `json:"stream,omitempty"`
}

type BenchmarkClient 

type BenchmarkClient struct {
	// contains filtered or unexported fields
}

func NewBenchmarkClient 

func NewBenchmarkClient(opts *BenchmarkOptions) *BenchmarkClient

func (*BenchmarkClient) TestChatEndpoint 

func (bc *BenchmarkClient) TestChatEndpoint(model string, messages []map[string]interface{}, concurrency int, totalRequests int) (*BenchmarkResult, error)

func (*BenchmarkClient) TestMessagesEndpoint 

func (bc *BenchmarkClient) TestMessagesEndpoint(model string, messages []map[string]interface{}, maxTokens int, concurrency int, totalRequests int) (*BenchmarkResult, error)

func (*BenchmarkClient) TestModelsEndpoint 

func (bc *BenchmarkClient) TestModelsEndpoint(concurrency int, totalRequests int) (*BenchmarkResult, error)

type BenchmarkOptions 

type BenchmarkOptions struct {
	BaseURL          string
	Timeout          time.Duration
	MaxConns         int
	Provider         string // "openai" or "anthropic"
	APIKey           string
	DisableKeepAlive bool
}

type BenchmarkResult 

type BenchmarkResult struct {
	TotalRequests    int           `json:"totalRequests"`
	SuccessRequests  int           `json:"successRequests"`
	FailedRequests   int           `json:"failedRequests"`
	TotalDuration    time.Duration `json:"totalDuration"`
	AvgResponseTime  time.Duration `json:"avgResponseTime"`
	MinResponseTime  time.Duration `json:"minResponseTime"`
	MaxResponseTime  time.Duration `json:"maxResponseTime"`
	RequestsPerSec   float64       `json:"requestsPerSec"`
	TotalBytes       int64         `json:"totalBytes"`
	ErrorRate        float64       `json:"errorRate"`
	StatusCodeCounts map[int]int   `json:"statusCodeCounts"`
}

func (*BenchmarkResult) PrintSummary 

func (br *BenchmarkResult) PrintSummary()

type MockServer 

type MockServer struct {
	// contains filtered or unexported fields
}

MockServer represents the mock server

Example 
// Create a mock server without authentication (default behavior)
server := NewMockServer(
	WithPort(8082),
	WithOpenAIDefaults(),
	// No WithapiKey called - all requests allowed
)

server.Start()

func NewMockServer 

func NewMockServer(opts ...Option) *MockServer

NewMockServer creates a new mock server with the given options

func (*MockServer) Port 

func (ms *MockServer) Port() int

Port returns the configured port

func (*MockServer) Start 

func (ms *MockServer) Start() error

Start starts the mock server

func (*MockServer) Stop 

func (ms *MockServer) Stop() error

Stop stops the mock server

func (*MockServer) UseAuthMiddleware 

func (ms *MockServer) UseAuthMiddleware()

UseAuthMiddleware applies the auth middleware to all routes

Example 
// Create a mock server
server := NewMockServer(
	WithPort(8083),
	WithApiKey("my-secret-key"),
)

// Manually apply auth middleware (useful if you want to apply it conditionally)
server.UseAuthMiddleware()

server.Start()

type Model 

type Model struct {
	ID      string `json:"id"`
	Object  string `json:"object"`
	Created int64  `json:"created"`
	OwnedBy string `json:"owned_by"`
}

Model represents a model in the models list

type OpenAIChatRequest 

type OpenAIChatRequest struct {
	Model    string                   `json:"model"`
	Messages []map[string]interface{} `json:"messages"`
	Stream   bool                     `json:"stream,omitempty"`
}

Use simple structs for requests instead of importing from SDKs

type Option 

type Option func(*serverConfig)

Option is a functional option for configuring the MockServer

func WithAnthropicDefaults 

func WithAnthropicDefaults() Option

WithAnthropicDefaults sets defaults for Anthropic endpoints

func WithApiKey 

func WithApiKey(key string) Option

WithApiKey sets the authentication key required for requests

Example 
// Create a mock server with authentication
server := NewMockServer(
	WithPort(8081),
	WithApiKey("test-api-key-123"),
	WithOpenAIDefaults(),
)

// The auth middleware will be automatically applied when the server starts
// Clients must provide either:
// - Authorization: Bearer test-api-key-123
// - x-api-key: test-api-key-123
//
// If WithapiKey is not called or the key is empty, no authentication is required.

server.Start()

func WithBothDefaults 

func WithBothDefaults() Option

WithBothDefaults sets defaults for both OpenAI and Anthropic

func WithChatDelay 

func WithChatDelay(delayMs int) Option

WithChatDelay sets a fixed delay for chat responses (in milliseconds)

func WithChatResponse 

func WithChatResponse(response json.RawMessage) Option

WithChatResponse adds a single chat completion response (as raw JSON)

func WithChatResponseContent 

func WithChatResponseContent(content string) Option

WithChatResponseContent adds a chat response from content string

func WithChatResponses 

func WithChatResponses(responses []json.RawMessage, loop bool) Option

WithChatResponses sets the default chat completion responses

func WithDefaultModels 

func WithDefaultModels(models []Model) Option

WithDefaultModels sets the default model list

func WithDefaultOptions 

func WithDefaultOptions() Option

WithDefaultOptions applies sensible defaults

func WithMessageDelay 

func WithMessageDelay(delayMs int) Option

WithMessageDelay sets a fixed delay for message responses (in milliseconds)

func WithMessageResponse 

func WithMessageResponse(response json.RawMessage) Option

WithMessageResponse adds a single message response (as raw JSON)

func WithMessageResponseContent 

func WithMessageResponseContent(content string) Option

WithMessageResponseContent adds a message response from content string

func WithMessageResponses 

func WithMessageResponses(responses []json.RawMessage, loop bool) Option

WithMessageResponses sets the default message responses

func WithOpenAIDefaults 

func WithOpenAIDefaults() Option

WithOpenAIDefaults sets defaults for OpenAI endpoints

func WithPort 

func WithPort(port int) Option

WithPort sets the server port

func WithRandomDelay 

func WithRandomDelay(minMs, maxMs int) Option

WithRandomDelay sets a random delay range (in milliseconds)

type RequestResult 

type RequestResult struct {
	Duration   time.Duration
	StatusCode int
	Error      error
	Bytes      int64
}

Source Files

View all Source files

Directories

Path Synopsis
examples
client command
server command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.