llmux

README

LLMux

English | Simplified Chinese

LLMux is a high-performance LLM gateway written in Go. It supports standalone deployments and distributed, enterprise-grade governance with optional Postgres and Redis. A Next.js dashboard provides management and analytics when enabled.

Overview

• Unified OpenAI-compatible APIs: chat, responses, embeddings, models
• Multi-provider routing with six strategies (shuffle, round-robin, lowest-latency, least-busy, lowest-tpm-rpm, lowest-cost)
• Governance: multi-tenant auth, budgets, rate limits, audit logging
• Ops-friendly: Prometheus metrics, OpenTelemetry tracing, health checks
• Optional Next.js dashboard for management and analytics

Performance: LLMux vs LiteLLM

We benchmarked LLMux (Go) against LiteLLM (Python) on identical hardware (4 CPU cores) using a local mock server with fixed 50ms latency.

Metric	LLMux (Go)	LiteLLM (Python)	Difference
Throughput (RPS)	1943.35	246.52	~8x Faster
Mean Latency	51.29 ms	403.94 ms	~8x Lower Overhead
P99 Latency	91.71 ms	845.37 ms	Stable vs Jittery

Benchmark config: 10k requests, 100 concurrency, 4 CPU cores, 50ms backend latency.

Quick Start

Prerequisites

• Go 1.23+
• Node.js 18+ (dashboard)
• Optional: PostgreSQL for auth/usage tracking
• Optional: Redis for distributed routing + rate limiting

Build and Run

git clone https://github.com/blueberrycongee/llmux.git
cd llmux

cp .env.example .env
# Edit .env with your API keys

make build
cp config/config.example.yaml config/config.yaml
./bin/llmux --config config/config.yaml

Run Dashboard

cd ui
npm install
npm run dev

Docker

docker build -t llmux .
docker run -p 8080:8080 -v $(pwd)/config:/config llmux

Configuration

Environment Variables

# Provider API Keys
OPENAI_API_KEY=sk-xxx
ANTHROPIC_API_KEY=sk-ant-xxx
GOOGLE_API_KEY=xxx
AZURE_OPENAI_API_KEY=xxx

# Database (optional, enables enterprise features)
DB_HOST=localhost
DB_USER=llmux
DB_PASSWORD=xxx
DB_NAME=llmux

# Redis (optional, for distributed caching)
REDIS_ADDR=localhost:6379
REDIS_PASSWORD=xxx

# Dashboard
NEXT_PUBLIC_API_URL=http://localhost:8080

config.yaml (minimal)

server:
  port: 8080
  admin_port: 0
  read_timeout: 30s
  write_timeout: 120s

deployment:
  mode: standalone  # standalone, distributed, development

providers:
  - name: openai
    type: openai
    api_key: ${OPENAI_API_KEY}
    base_url: https://api.openai.com/v1
    models:
      - gpt-4o
      - gpt-4o-mini

routing:
  strategy: simple-shuffle
  fallback_enabled: true
  retry_count: 3
  distributed: false

metrics:
  enabled: true
  path: /metrics

Note: auth.enabled defaults to true for non-development modes. For unauthenticated local testing, set deployment.mode=development and auth.enabled=false.

Deployment Modes

• standalone: in-memory state, intended for single-instance runs.
• distributed: requires PostgreSQL for auth/usage state and Redis for routing stats and rate limiting.
• development: allows in-memory state for multi-instance testing (not consistent).

Routing Strategies

Strategy	Description
simple-shuffle	Random selection with optional weight/rpm/tpm weighting
round-robin	Cycles through deployments, Redis-backed when distributed
lowest-latency	Selects deployment with lowest average latency (streaming-aware)
least-busy	Selects deployment with fewest active requests
lowest-tpm-rpm	Selects deployment with lowest TPM/RPM usage
lowest-cost	Selects deployment with lowest cost per token

API Reference

Chat Completions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'

Responses

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "model": "gpt-4o",
    "input": "Hello!"
  }'

List Models

curl http://localhost:8080/v1/models

Health Check

curl http://localhost:8080/health/live
curl http://localhost:8080/health/ready

Management API

Management endpoints (and the dashboard UI) are exposed only when server.admin_port > 0 is configured. Database configuration controls whether management state is persisted (Postgres) or ephemeral (in-memory).

Bootstrap (optional): set auth.bootstrap_token and pass X-LLMux-Bootstrap-Token to access management endpoints for initial setup.

Key categories:

• Keys: /key/*
• Users: /user/*
• Teams: /team/*
• Organizations: /organization/*
• Spend/usage: /spend/*, /global/*
• Audit: /audit/*
• Control: /control/*

Operations and Observability

• Metrics: Prometheus at metrics.path (default /metrics)
• Tracing: OpenTelemetry exporter configuration via tracing.* settings
• Logs: structured JSON logs from the gateway and management APIs
• Auditing: append-only audit logs when the audit store is configured

Production Notes

• Standalone mode is single-node and uses in-memory state only.
• Distributed mode requires Postgres for auth/usage and Redis for routing stats and rate limiting; missing dependencies degrade related features.
• /v1/audio/* and /v1/batches currently return invalid_request_error until provider support is implemented.

Project Structure

llmux/
|-- cmd/server/           # Gateway entry point
|-- config/               # Configuration files
|-- internal/
|   |-- api/              # HTTP handlers & management endpoints
|   |-- auth/             # Authentication, authorization & stores
|   |-- cache/            # Response caching (local/redis/dual)
|   |-- config/           # Configuration loading
|   |-- metrics/          # Prometheus & OpenTelemetry
|   `-- router/           # Request routing strategies
|-- providers/            # LLM provider adapters
|-- pkg/
|   |-- types/            # Shared types
|   `-- errors/           # Error definitions
|-- ui/                   # Next.js Dashboard
|-- deploy/               # Deployment configs
|-- bench/                # Benchmark tools
`-- tests/                # Integration tests

Developer Info

Documentation

• docs/DEVELOPMENT.md
• docs/PRODUCTION_TEST_GUIDE.md
• docs/runbooks/DISTRIBUTED_MODE.md

Development Commands

make test
make coverage
make lint
make fmt
make check

Contributing

See CONTRIBUTING.md for contribution guidelines.

License

MIT License - see LICENSE

Documentation

Overview

Package llmux provides a high-performance LLM gateway as a Go library. It supports 44+ LLM providers with unified OpenAI-compatible API.

LLMux can be used in two modes:

• Library Mode: Import and use directly in your Go application
• Gateway Mode: Run as a standalone HTTP proxy server

Basic usage:

client, err := llmux.New(
    llmux.WithProvider(llmux.ProviderConfig{
        Name:   "openai",
        Type:   "openai",
        APIKey: os.Getenv("OPENAI_API_KEY"),
        Models: []string{"gpt-4o", "gpt-4o-mini"},
    }),
)
if err != nil {
    log.Fatal(err)
}
defer client.Close()

resp, err := client.ChatCompletion(ctx, &llmux.ChatRequest{
    Model: "gpt-4o",
    Messages: []llmux.ChatMessage{
        {Role: "user", Content: json.RawMessage(`"Hello!"`)},
    },
})

Index

• Constants
• Variables
• func WithRateLimitAPIKey(ctx context.Context, apiKey string) context.Context
• type Cache
• type CacheControl
• type CacheEntry
• type CacheStats
• type CacheType
• type ChatMessage
• type ChatRequest
• type ChatResponse
• type Choice
• type Client
  • func New(opts ...Option) (*Client, error)
  • func (c *Client) AddProvider(name string, prov Provider, models []string) error
  • func (c *Client) CalculateCost(model string, usage *types.Usage) float64
  • func (c *Client) ChatCompletion(ctx context.Context, req *ChatRequest) (*ChatResponse, error)
  • func (c *Client) ChatCompletionStream(ctx context.Context, req *ChatRequest) (*StreamReader, error)
  • func (c *Client) Close() error
  • func (c *Client) Embedding(ctx context.Context, req *types.EmbeddingRequest) (*types.EmbeddingResponse, error)
  • func (c *Client) GetProvider(name string) (Provider, bool)
  • func (c *Client) GetProviders() []string
  • func (c *Client) GetStats(deploymentID string) *DeploymentStats
  • func (c *Client) ListDeployments() []*provider.Deployment
  • func (c *Client) ListModels(ctx context.Context) ([]Model, error)
  • func (c *Client) RegisterProviderFactory(providerType string, factory ProviderFactory)
  • func (c *Client) RemoveProvider(name string) error
  • func (c *Client) ResilienceStats(key string) ResilienceStats
  • func (c *Client) SetCooldown(deploymentID string, until time.Time) error
• type ClientConfig
• type Deployment
• type DeploymentConfig
• type DeploymentStats
• type EmbeddingData
• type EmbeddingRequest
• type EmbeddingResponse
• type EmbeddingUsage
• type FallbackReporter
• type LLMError
• type Model
• type Option
  • func WithCache(cache Cache) Option
  • func WithCacheTTL(ttl time.Duration) Option
  • func WithCacheTypeLabel(label string) Option
  • func WithCooldown(d time.Duration) Option
  • func WithDefaultProvider(provider string) Option
  • func WithEWMAAlpha(alpha float64) Option
  • func WithFallback(enabled bool) Option
  • func WithFallbackReporter(reporter FallbackReporter) Option
  • func WithLogger(logger *slog.Logger) Option
  • func WithOTelMetrics(config observability.OTelMetricsConfig) Option
  • func WithPlugin(p plugin.Plugin) Option
  • func WithPluginConfig(config plugin.PipelineConfig) Option
  • func WithPricingFile(path string) Option
  • func WithProvider(cfg ProviderConfig) Option
  • func WithProviderInstance(name string, prov Provider, models []string) Option
  • func WithRateLimiter(limiter resilience.DistributedLimiter) Option
  • func WithRateLimiterConfig(config RateLimiterConfig) Option
  • func WithRetry(count int, backoff time.Duration) Option
  • func WithRetryJitter(jitter float64) Option
  • func WithRetryMaxBackoff(d time.Duration) Option
  • func WithRoundRobinStore(store router.RoundRobinStore) Option
  • func WithRouter(r Router) Option
  • func WithRouterStrategy(strategy Strategy) Option
  • func WithStatsStore(store router.StatsStore) Option
  • func WithStreamRecoveryMaxAccumulatedBytes(maxBytes int) Option
  • func WithStreamRecoveryMode(mode StreamRecoveryMode) Option
  • func WithTimeout(d time.Duration) Option
• type PluginContext
• type PluginInterface
• type PluginPipelineConfig
• type PluginShortCircuit
• type Provider
• type ProviderConfig
• type ProviderFactory
• type RateLimitKeyStrategy
• type RateLimiterConfig
• type RequestContext
• type ResilienceStats
• type ResponseFormat
• type ResponseMetrics
• type Router
• type RouterConfig
• type Strategy
• type StreamChoice
• type StreamChunk
• type StreamDelta
• type StreamOptions
• type StreamReader
  • func (s *StreamReader) Close() error
  • func (s *StreamReader) Recv() (*types.StreamChunk, error)
  • func (s *StreamReader) TTFT() time.Duration
• type StreamRecoveryMode
• type Tool
• type ToolCall
• type ToolCallFunction
• type ToolFunction
• type Usage

Constants

const (
	// StrategySimpleShuffle randomly selects from available deployments.
	StrategySimpleShuffle = router.StrategySimpleShuffle

	// StrategyShuffle is an alias for StrategySimpleShuffle.
	StrategyShuffle = router.StrategySimpleShuffle

	// StrategyRoundRobin selects deployments in strict round-robin order.
	StrategyRoundRobin = router.StrategyRoundRobin

	// StrategyLowestLatency selects the deployment with lowest average latency.
	StrategyLowestLatency = router.StrategyLowestLatency

	// StrategyLeastBusy selects the deployment with fewest active requests.
	StrategyLeastBusy = router.StrategyLeastBusy

	// StrategyLowestTPMRPM selects the deployment with lowest TPM/RPM usage.
	StrategyLowestTPMRPM = router.StrategyLowestTPMRPM

	// StrategyLowestCost selects the deployment with lowest cost per token.
	StrategyLowestCost = router.StrategyLowestCost

	// StrategyTagBased filters deployments based on request tags.
	StrategyTagBased = router.StrategyTagBased
)

Re-export routing strategy constants.

const (
	// CacheTypeLocal is an in-memory cache.
	CacheTypeLocal = cache.TypeLocal

	// CacheTypeRedis is a Redis cache.
	CacheTypeRedis = cache.TypeRedis

	// CacheTypeDual is an in-memory + Redis dual cache.
	CacheTypeDual = cache.TypeDual

	// CacheTypeSemantic is a semantic cache with vector similarity.
	CacheTypeSemantic = cache.TypeSemantic
)

Re-export cache type constants.

const (
	TypeAuthentication     = errors.TypeAuthentication
	TypeRateLimit          = errors.TypeRateLimit
	TypeInvalidRequest     = errors.TypeInvalidRequest
	TypeNotFound           = errors.TypeNotFound
	TypeTimeout            = errors.TypeTimeout
	TypeServiceUnavailable = errors.TypeServiceUnavailable
	TypeInternalError      = errors.TypeInternalError
	TypeContextLength      = errors.TypeContextLength
	TypeContentPolicy      = errors.TypeContentPolicy
	TypePermissionDenied   = errors.TypePermissionDenied
	TypeInsufficientQuota  = errors.TypeInsufficientQuota
)

Re-export error type constants.

const Version = "2.0.0"

Version is the current version of LLMux.

Variables

var (
	NewAuthenticationError     = errors.NewAuthenticationError
	NewPermissionError         = errors.NewPermissionError
	NewRateLimitError          = errors.NewRateLimitError
	NewInsufficientQuotaError  = errors.NewInsufficientQuotaError
	NewInvalidRequestError     = errors.NewInvalidRequestError
	NewNotFoundError           = errors.NewNotFoundError
	NewTimeoutError            = errors.NewTimeoutError
	NewServiceUnavailableError = errors.NewServiceUnavailableError
	NewInternalError           = errors.NewInternalError
)

Re-export error factory functions.

Functions

func WithRateLimitAPIKey

func WithRateLimitAPIKey(ctx context.Context, apiKey string) context.Context

WithRateLimitAPIKey stores an API key identifier in the context for rate limiting. This is only used to derive the rate limit key and is never sent to providers.

Types

type Cache

type Cache = cache.Cache

Cache defines the interface for all cache implementations.

type CacheControl

type CacheControl = cache.Control

CacheControl allows per-request cache behavior customization.

type CacheEntry

type CacheEntry = cache.Entry

CacheEntry represents a single cache entry for pipeline operations.

type CacheStats

type CacheStats = cache.Stats

CacheStats holds cache statistics for monitoring.

type CacheType

type CacheType = cache.Type

CacheType represents the type of cache backend.

type ChatMessage

type ChatMessage = types.ChatMessage

ChatMessage represents a single message in the conversation.

type ChatRequest

type ChatRequest = types.ChatRequest

ChatRequest represents an OpenAI-compatible chat completion request.

type ChatResponse

type ChatResponse = types.ChatResponse

ChatResponse represents an OpenAI-compatible chat completion response.

type Choice

type Choice = types.Choice

Choice represents a single completion choice.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is the main entry point for LLMux library mode. It manages providers, routing, caching, and request execution.

Client is safe for concurrent use by multiple goroutines.

func New

func New(opts ...Option) (*Client, error)

New creates a new LLMux client with the given options.

Example:

client, err := llmux.New(
    llmux.WithProvider(llmux.ProviderConfig{
        Name:   "openai",
        Type:   "openai",
        APIKey: os.Getenv("OPENAI_API_KEY"),
        Models: []string{"gpt-4o"},
    }),
    llmux.WithRouterStrategy(llmux.StrategyLowestLatency),
)

func (*Client) AddProvider

func (c *Client) AddProvider(name string, prov Provider, models []string) error

AddProvider adds a provider at runtime. This is useful for dynamically adding providers without recreating the client.

func (*Client) CalculateCost

func (c *Client) CalculateCost(model string, usage *types.Usage) float64

CalculateCost computes the usage cost for a given model using loaded pricing data. Returns 0 when pricing is unavailable or model not found.

func (*Client) ChatCompletion

func (c *Client) ChatCompletion(ctx context.Context, req *ChatRequest) (*ChatResponse, error)

ChatCompletion sends a chat completion request. It handles routing, retries, fallback, and caching automatically.

func (*Client) ChatCompletionStream

func (c *Client) ChatCompletionStream(ctx context.Context, req *ChatRequest) (*StreamReader, error)

ChatCompletionStream sends a streaming chat completion request. Returns a StreamReader that can be used to iterate over response chunks.

Example:

stream, err := client.ChatCompletionStream(ctx, req)
if err != nil {
    return err
}
defer stream.Close()

for {
    chunk, err := stream.Recv()
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
    fmt.Print(chunk.Choices[0].Delta.Content)
}

func (*Client) Close

func (c *Client) Close() error

Close releases all resources held by the client.

func (*Client) Embedding

func (c *Client) Embedding(ctx context.Context, req *types.EmbeddingRequest) (*types.EmbeddingResponse, error)

Embedding sends an embedding request. It handles routing, retries, and fallback automatically.

func (*Client) GetProvider

func (c *Client) GetProvider(name string) (Provider, bool)

GetProvider returns the provider instance by name.

func (*Client) GetProviders

func (c *Client) GetProviders() []string

GetProviders returns the names of all registered providers.

func (*Client) GetStats

func (c *Client) GetStats(deploymentID string) *DeploymentStats

GetStats returns routing statistics for a deployment.

func (*Client) ListDeployments

func (c *Client) ListDeployments() []*provider.Deployment

ListDeployments returns a snapshot of all deployments.

func (*Client) ListModels

func (c *Client) ListModels(ctx context.Context) ([]Model, error)

ListModels returns all available models from registered providers.

func (*Client) RegisterProviderFactory

func (c *Client) RegisterProviderFactory(providerType string, factory ProviderFactory)

RegisterProviderFactory registers a custom provider factory. This allows adding support for new provider types.

func (*Client) RemoveProvider

func (c *Client) RemoveProvider(name string) error

RemoveProvider removes a provider at runtime.

func (*Client) ResilienceStats

func (c *Client) ResilienceStats(key string) ResilienceStats

ResilienceStats returns the resilience status for a provider key.

func (*Client) SetCooldown

func (c *Client) SetCooldown(deploymentID string, until time.Time) error

SetCooldown updates the cooldown expiration time for a deployment. A zero time clears any active cooldown.

type ClientConfig

type ClientConfig struct {
	// Providers configuration
	Providers []ProviderConfig

	// Custom provider instances (for advanced use)
	ProviderInstances []providerInstance

	// Routing
	RouterStrategy   Strategy
	Router           Router // Custom router (overrides RouterStrategy)
	FallbackEnabled  bool
	RetryCount       int
	RetryBackoff     time.Duration
	RetryMaxBackoff  time.Duration
	RetryJitter      float64
	CooldownPeriod   time.Duration
	EWMAAlpha        float64
	DefaultProvider  string
	FallbackReporter FallbackReporter

	// Distributed Routing Stats (for multi-instance deployments)
	StatsStore router.StatsStore

	// Distributed Round Robin counters (for multi-instance deployments)
	RoundRobinStore router.RoundRobinStore

	// Caching
	CacheEnabled   bool
	Cache          Cache // Custom cache implementation
	CacheTTL       time.Duration
	CacheTypeLabel string

	// HTTP
	Timeout time.Duration

	// Logging
	Logger *slog.Logger

	// Plugins
	Plugins      []plugin.Plugin
	PluginConfig *plugin.PipelineConfig

	// Pricing
	PricingFile string

	// Stream recovery
	StreamRecoveryMode StreamRecoveryMode
	// StreamRecoveryMaxAccumulatedBytes caps the in-memory accumulated stream content used for recovery.
	// Set to 0 to disable the cap (not recommended).
	StreamRecoveryMaxAccumulatedBytes int

	// Observability
	OTelMetricsConfig observability.OTelMetricsConfig

	// Rate Limiting (Distributed)
	RateLimiter       resilience.DistributedLimiter
	RateLimiterConfig RateLimiterConfig
}

ClientConfig holds all configuration for the LLMux client.

type Deployment

type Deployment = provider.Deployment

Deployment represents a specific model deployment configuration.

type DeploymentConfig

type DeploymentConfig = router.DeploymentConfig

DeploymentConfig contains deployment-specific configuration for routing.

type DeploymentStats

type DeploymentStats = router.DeploymentStats

DeploymentStats tracks performance metrics for a deployment.

type EmbeddingData

type EmbeddingData struct {
	Object    string    `json:"object"`
	Embedding []float64 `json:"embedding"`
	Index     int       `json:"index"`
}

EmbeddingData represents a single embedding vector.

type EmbeddingRequest

type EmbeddingRequest struct {
	Model string   `json:"model"`
	Input []string `json:"input"`
}

EmbeddingRequest represents an embedding request (future support).

type EmbeddingResponse

type EmbeddingResponse struct {
	Object string          `json:"object"`
	Data   []EmbeddingData `json:"data"`
	Model  string          `json:"model"`
	Usage  *EmbeddingUsage `json:"usage,omitempty"`
}

EmbeddingResponse represents an embedding response (future support).

type EmbeddingUsage

type EmbeddingUsage struct {
	PromptTokens int `json:"prompt_tokens"`
	TotalTokens  int `json:"total_tokens"`
}

EmbeddingUsage contains token usage for embedding requests.

type FallbackReporter

type FallbackReporter func(ctx context.Context, originalModel, fallbackModel string, err error, success bool)

FallbackReporter receives fallback outcomes for observability.

type LLMError

type LLMError = errors.LLMError

LLMError represents a standardized error from an LLM provider.

type Model

type Model struct {
	ID       string `json:"id"`
	Provider string `json:"provider"`
	Object   string `json:"object,omitempty"`
}

Model represents an available model from a provider.

type Option

type Option func(*ClientConfig)

Option is a function that configures the Client.

func WithCache

func WithCache(cache Cache) Option

WithCache enables caching with the given implementation.

Example:

redisCache, _ := redis.New(redis.WithAddr("localhost:6379"))
llmux.WithCache(redisCache)

func WithCacheTTL

func WithCacheTTL(ttl time.Duration) Option

WithCacheTTL sets the default cache TTL. This is used when no TTL is specified in the cache control.

func WithCacheTypeLabel

func WithCacheTypeLabel(label string) Option

WithCacheTypeLabel sets the cache type label for metrics.

func WithCooldown

func WithCooldown(d time.Duration) Option

WithCooldown sets the cooldown period for failed deployments. Deployments that fail will be excluded from routing for this duration.

func WithDefaultProvider

func WithDefaultProvider(provider string) Option

WithDefaultProvider prefers a provider when multiple deployments are available.

func WithEWMAAlpha

func WithEWMAAlpha(alpha float64) Option

WithEWMAAlpha sets the smoothing factor for EWMA calculations. alpha should be between 0 and 1. A higher alpha discounts older observations faster.

func WithFallback

func WithFallback(enabled bool) Option

WithFallback enables/disables fallback on failure. When enabled, failed requests will be retried on different deployments.

func WithFallbackReporter

func WithFallbackReporter(reporter FallbackReporter) Option

WithFallbackReporter records fallback outcomes.

func WithLogger

func WithLogger(logger *slog.Logger) Option

WithLogger sets the logger for the client. The logger is used for debug, info, and error messages.

func WithOTelMetrics

func WithOTelMetrics(config observability.OTelMetricsConfig) Option

WithOTelMetrics configures OpenTelemetry metrics.

func WithPlugin

func WithPlugin(p plugin.Plugin) Option

WithPlugin adds a plugin to the client. Plugins are executed in the order of their priority.

func WithPluginConfig

func WithPluginConfig(config plugin.PipelineConfig) Option

WithPluginConfig sets the plugin pipeline configuration.

func WithPricingFile

func WithPricingFile(path string) Option

WithPricingFile sets the path to the custom pricing JSON file.

func WithProvider

func WithProvider(cfg ProviderConfig) Option

WithProvider adds a provider configuration. The provider will be created automatically based on the Type field.

Example:

llmux.WithProvider(llmux.ProviderConfig{
    Name:   "openai",
    Type:   "openai",
    APIKey: os.Getenv("OPENAI_API_KEY"),
    Models: []string{"gpt-4o", "gpt-4o-mini"},
})

func WithProviderInstance

func WithProviderInstance(name string, prov Provider, models []string) Option

WithProviderInstance adds a pre-configured provider instance. Use this when you need custom provider configuration beyond what ProviderConfig offers.

Example:

openaiProvider := openai.New(
    openai.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    openai.WithBaseURL("https://custom-endpoint.com/v1"),
)
llmux.WithProviderInstance("my-openai", openaiProvider, []string{"gpt-4o"})

func WithRateLimiter

func WithRateLimiter(limiter resilience.DistributedLimiter) Option

WithRateLimiter sets a custom distributed rate limiter implementation. This enables distributed rate limiting across multiple LLMux instances.

Example:

redisClient := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
limiter := resilience.NewRedisLimiter(redisClient)
llmux.WithRateLimiter(limiter)

func WithRateLimiterConfig

func WithRateLimiterConfig(config RateLimiterConfig) Option

WithRateLimiterConfig sets the rate limiter configuration. This configures the rate limits (RPM/TPM) and key strategy.

Example:

llmux.WithRateLimiterConfig(llmux.RateLimiterConfig{
    Enabled:     true,
    RPMLimit:    100,
    TPMLimit:    10000,
    WindowSize:  time.Minute,
    KeyStrategy: llmux.RateLimitKeyByAPIKey,
})

func WithRetry

func WithRetry(count int, backoff time.Duration) Option

WithRetry configures retry behavior. count: number of retry attempts (0 = no retries) backoff: initial backoff duration (exponential backoff is applied)

func WithRetryJitter

func WithRetryJitter(jitter float64) Option

WithRetryJitter sets the jitter ratio for retries (0.0 - 1.0).

func WithRetryMaxBackoff

func WithRetryMaxBackoff(d time.Duration) Option

WithRetryMaxBackoff sets the maximum backoff duration for retries. Use 0 to disable the cap.

func WithRoundRobinStore

func WithRoundRobinStore(store router.RoundRobinStore) Option

WithRoundRobinStore sets a distributed round-robin store for consistent RR across instances.

func WithRouter

func WithRouter(r Router) Option

WithRouter sets a custom router implementation. This overrides the RouterStrategy option.

Example:

customRouter := myrouter.New(...)
llmux.WithRouter(customRouter)

func WithRouterStrategy

func WithRouterStrategy(strategy Strategy) Option

WithRouterStrategy sets the routing strategy. Available strategies:

• StrategySimpleShuffle: Random selection with optional weights
• StrategyLowestLatency: Select deployment with lowest latency
• StrategyLeastBusy: Select deployment with fewest active requests
• StrategyLowestTPMRPM: Select deployment with lowest token/request usage
• StrategyLowestCost: Select deployment with lowest cost
• StrategyTagBased: Filter deployments by tags

func WithStatsStore

func WithStatsStore(store router.StatsStore) Option

WithStatsStore sets a distributed stats store for routing decisions. This enables multi-instance deployments to share routing statistics (latency, active requests, cooldowns), making load-balanced strategies like least-busy and lowest-latency work correctly across a cluster.

Example:

redisClient := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
statsStore := router.NewRedisStatsStore(redisClient)
llmux.WithStatsStore(statsStore)

func WithStreamRecoveryMaxAccumulatedBytes

func WithStreamRecoveryMaxAccumulatedBytes(maxBytes int) Option

WithStreamRecoveryMaxAccumulatedBytes caps the accumulated stream content retained in-memory for recovery. A value of 0 disables the cap.

func WithStreamRecoveryMode

func WithStreamRecoveryMode(mode StreamRecoveryMode) Option

WithStreamRecoveryMode configures how streaming recovery behaves after a mid-stream failure.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout sets the HTTP request timeout. This applies to all provider API calls.

type PluginContext

type PluginContext = plugin.Context

PluginContext provides execution context for plugins.

type PluginInterface

type PluginInterface = plugin.Plugin

PluginInterface defines the interface for LLMux plugins. Plugins can intercept and modify requests/responses at various lifecycle points.

type PluginPipelineConfig

type PluginPipelineConfig = plugin.PipelineConfig

PluginPipelineConfig holds configuration for the plugin pipeline.

type PluginShortCircuit

type PluginShortCircuit = plugin.ShortCircuit

PluginShortCircuit represents a plugin's decision to short-circuit the request.

type Provider

type Provider = provider.Provider

Provider defines the interface that all LLM provider adapters must implement.

type ProviderConfig

type ProviderConfig = provider.Config

ProviderConfig contains provider-specific configuration.

type ProviderFactory

type ProviderFactory = provider.Factory

ProviderFactory creates provider instances from configuration.

type RateLimitKeyStrategy

type RateLimitKeyStrategy string

RateLimitKeyStrategy defines how to derive the rate limit key.

const (
	// RateLimitKeyByAPIKey uses the API key as the rate limit key.
	RateLimitKeyByAPIKey RateLimitKeyStrategy = "api_key" // #nosec G101 -- identifier value, not a credential.
	// RateLimitKeyByUser uses the user ID as the rate limit key.
	RateLimitKeyByUser RateLimitKeyStrategy = "user"
	// RateLimitKeyByModel uses the model name as the rate limit key.
	RateLimitKeyByModel RateLimitKeyStrategy = "model"
	// RateLimitKeyByAPIKeyAndModel uses both API key and model as the rate limit key.
	RateLimitKeyByAPIKeyAndModel RateLimitKeyStrategy = "api_key_model" // #nosec G101 -- identifier value, not a credential.
)

type RateLimiterConfig

type RateLimiterConfig struct {
	// Enabled indicates whether rate limiting is enabled.
	Enabled bool
	// RPMLimit is the requests per minute limit.
	RPMLimit int64
	// TPMLimit is the tokens per minute limit.
	TPMLimit int64
	// WindowSize is the sliding window duration (default: 1 minute).
	WindowSize time.Duration
	// KeyStrategy defines how to derive the rate limit key.
	KeyStrategy RateLimitKeyStrategy
	// FailOpen allows requests when the rate limiter backend fails.
	FailOpen bool
}

RateLimiterConfig holds configuration for rate limiting.

type RequestContext

type RequestContext = router.RequestContext

RequestContext contains request-specific information for routing decisions.

type ResilienceStats

type ResilienceStats struct {
	Key                string  `json:"key"`
	CircuitState       string  `json:"circuit_state"`
	RateLimitTokens    float64 `json:"rate_limit_tokens"`
	ConcurrentCurrent  int     `json:"concurrent_current"`
	ConcurrentCapacity int     `json:"concurrent_capacity"`
}

ResilienceStats exposes provider-level resilience metrics.

type ResponseFormat

type ResponseFormat = types.ResponseFormat

ResponseFormat specifies the output format for the model.

type ResponseMetrics

type ResponseMetrics = router.ResponseMetrics

ResponseMetrics contains metrics from a completed request.

type Router

type Router = router.Router

Router selects the best deployment for a given request.

type RouterConfig

type RouterConfig = router.Config

RouterConfig contains router configuration options.

type Strategy

type Strategy = router.Strategy

Strategy defines the routing strategy type.

type StreamChoice

type StreamChoice = types.StreamChoice

StreamChoice represents a choice in a streaming response.

type StreamChunk

type StreamChunk = types.StreamChunk

StreamChunk represents a single chunk in a streaming response.

type StreamDelta

type StreamDelta = types.StreamDelta

StreamDelta contains the incremental content in a stream chunk.

type StreamOptions

type StreamOptions = types.StreamOptions

StreamOptions specifies options for streaming responses.

type StreamReader

type StreamReader struct {
	// contains filtered or unexported fields
}

StreamReader provides an iterator interface for streaming responses. It handles SSE parsing and provides a simple Recv() method for consuming chunks.

Example:

stream, err := client.ChatCompletionStream(ctx, req)
if err != nil {
    return err
}
defer stream.Close()

for {
    chunk, err := stream.Recv()
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
    fmt.Print(chunk.Choices[0].Delta.Content)
}

func (*StreamReader) Close

func (s *StreamReader) Close() error

Close releases resources associated with the stream. It's safe to call Close multiple times.

func (*StreamReader) Recv

func (s *StreamReader) Recv() (*types.StreamChunk, error)

Recv returns the next chunk from the stream. Returns io.EOF when the stream is complete. Returns an error if the stream encounters an error.

func (*StreamReader) TTFT

func (s *StreamReader) TTFT() time.Duration

TTFT returns the Time To First Token duration. Returns 0 if no chunks have been received yet.

type StreamRecoveryMode

type StreamRecoveryMode string

StreamRecoveryMode controls how stream recovery behaves after a mid-stream failure.

const (
	StreamRecoveryOff    StreamRecoveryMode = "off"
	StreamRecoveryAppend StreamRecoveryMode = "append"
	StreamRecoveryRetry  StreamRecoveryMode = "retry"
)

type Tool

type Tool = types.Tool

Tool represents a function that the model can call.

type ToolCall

type ToolCall = types.ToolCall

ToolCall represents a function call made by the model.

type ToolCallFunction

type ToolCallFunction = types.ToolCallFunction

ToolCallFunction contains the function name and arguments.

type ToolFunction

type ToolFunction = types.ToolFunction

ToolFunction describes a callable function.

type Usage

type Usage = types.Usage

Usage contains token usage statistics for the request.