aigateway

README

Ferro Labs AI Gateway

High-performance AI gateway in Go. Route LLM requests across 29 providers via a single OpenAI-compatible API.

🔀 29 providers, 2,500+ models — one API
⚡ 13,925 RPS at 1,000 concurrent users
📦 Single binary, zero dependencies, 32 MB base memory

---

Quick Start

Get from zero to first request in under 2 minutes.

Option A — Binary (fastest)

curl -fsSL https://github.com/ferro-labs/ai-gateway/releases/download/v1.0.0/ferro-gw_linux_amd64 -o ferro-gw
chmod +x ferro-gw
./ferro-gw --config config.yaml

Option B — Docker

docker pull ghcr.io/ferro-labs/ai-gateway:v1.0.0
docker run -p 8080:8080 \
  -e OPENAI_API_KEY=sk-your-key \
  ghcr.io/ferro-labs/ai-gateway:v1.0.0

Option C — Go

go install github.com/ferro-labs/ai-gateway/cmd/ferrogw@v1.0.0

Minimal config

Create config.yaml:

strategy:
  mode: fallback

targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
  - virtual_key: anthropic

aliases:
  fast: gpt-4o-mini
  smart: claude-3-5-sonnet-20241022

First request

export OPENAI_API_KEY=sk-your-key

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]
  }' | jq

---

Why Ferro Labs

Most AI gateways are Python proxies that crack under load or JavaScript services that eat memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput — a single binary that routes LLM requests with predictable latency and minimal resource usage.

Feature	Ferro Labs	LiteLLM	Bifrost	Kong AI
Language	Go	Python	Go	Go/Lua
Single binary	✅	❌	✅	❌
Providers	29	100+	20+	10+
MCP support	✅	❌	✅	❌
Response cache	✅	✅	✅	❌ (paid)
Guardrails	✅	✅	❌	❌ (paid)
OSS license	Apache 2.0	MIT	Apache 2.0	Apache 2.0
Managed cloud	Coming Soon	✅	✅	✅

---

Performance

Benchmarked against Kong OSS, Bifrost, LiteLLM, and Portkey on GCP n2-standard-8 (8 vCPU, 32 GB RAM) using a 60ms fixed-latency mock upstream — results reflect gateway overhead only.

Ferro Labs Latency Profile

VU	RPS	p50	p99	Memory
50	813	61.3ms	64.1ms	36 MB
150	2,447	61.2ms	63.4ms	47 MB
300	4,890	61.2ms	64.4ms	72 MB
500	8,014	61.5ms	72.9ms	89 MB
1,000	13,925	68.1ms	111.9ms	135 MB

At 1,000 VU: 13,925 RPS, p50 overhead 8.1ms, memory 135 MB. No connection pool failures. No throughput ceiling.

How to Reproduce

git clone https://github.com/ferro-labs/ai-gateway-performance-benchmarks
cd ai-gateway-performance-benchmarks
make setup && make bench

Full methodology, raw results, and flamegraph analysis: ferro-labs/ai-gateway-performance-benchmarks

---

Features

🔀 Routing

• 8 routing strategies: single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional
• Provider failover with configurable retry policies and status code filters
• Per-request model aliases (fast → gpt-4o-mini, smart → claude-3-5-sonnet)

🔌 Providers (29)

OpenAI & Compatible	Anthropic & Google	Cloud & Enterprise	Open Source & Inference
OpenAI	Anthropic	AWS Bedrock	Ollama
Azure OpenAI	Google Gemini	Azure Foundry	Hugging Face
OpenRouter	Vertex AI	Databricks	Replicate
DeepSeek		Cloudflare Workers AI	Together AI
Perplexity			Fireworks
xAI (Grok)			DeepInfra
Mistral			NVIDIA NIM
Groq			SambaNova
Cohere			Novita AI
AI21			Cerebras
Moonshot / Kimi			Qwen / DashScope

🛡️ Guardrails & Plugins

• Word/phrase filtering — block sensitive terms before they reach providers
• Token and message limits — enforce max_tokens and max_messages per request
• Response caching — in-memory cache with configurable TTL and entry limits
• Rate limiting — global RPS plus per-API-key and per-user RPM limits
• Budget controls — per-API-key USD spend tracking with configurable token pricing
• Request logging — structured logs with optional SQLite/PostgreSQL persistence

⚡ Performance

• Per-provider HTTP connection pools with optimized settings
• sync.Pool for JSON marshaling buffers and streaming I/O
• Zero-allocation stream detection, async hook dispatch batching
• Single binary, ~32 MB base memory, linear scaling to 1,000+ VUs

🤖 MCP (Model Context Protocol)

• Agentic tool-call loop — the gateway drives tool_calls automatically
• Streamable HTTP transport (MCP 2025-11-25 spec)
• Tool filtering with allowed_tools and bounded max_call_depth
• Multiple MCP servers with cross-server tool deduplication

📊 Observability

• Prometheus metrics at /metrics
• Deep health checks at /health with per-provider status
• Structured JSON request logging with SQLite/PostgreSQL persistence
• Admin API with usage stats, request logs, and config history/rollback
• Built-in dashboard UI at /dashboard
• HTTP-level connection tracing with DNS, TLS, and first-byte latency

---

Examples

Integration examples for common use cases are in ferro-labs/ai-gateway-examples:

Example	Description
basic	Single chat completion to the first configured provider
fallback	Fallback strategy — try providers in order with retries
loadbalance	Weighted load balancing across targets (70/30 split)
with-guardrails	Built-in word-filter and max-token guardrail plugins
with-mcp	Local MCP server with tool-calling integration
embedded	Embed the gateway as an HTTP handler inside an existing server

---

Configuration

Full annotated example — copy to config.yaml and customize:

# Routing strategy
strategy:
  mode: fallback  # single | fallback | loadbalance | conditional
                  # least-latency | cost-optimized | content-based | ab-test

# Provider targets (tried in order for fallback mode)
targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
      initial_backoff_ms: 100
  - virtual_key: anthropic
    retry:
      attempts: 2
  - virtual_key: gemini

# Model aliases — resolved before routing
aliases:
  fast: gpt-4o-mini
  smart: claude-3-5-sonnet-20241022
  cheap: gemini-1.5-flash

# Plugins — executed in order at the configured stage
plugins:
  - name: word-filter
    type: guardrail
    stage: before_request
    enabled: true
    config:
      blocked_words: ["password", "secret"]
      case_sensitive: false

  - name: max-token
    type: guardrail
    stage: before_request
    enabled: true
    config:
      max_tokens: 4096
      max_messages: 50

  - name: rate-limit
    type: guardrail
    stage: before_request
    enabled: true
    config:
      requests_per_second: 100
      key_rpm: 60

  - name: request-logger
    type: logging
    stage: before_request
    enabled: true
    config:
      level: info
      persist: true
      backend: sqlite
      dsn: ferrogw-requests.db

# MCP tool servers (optional)
mcp_servers:
  - name: my-tools
    url: https://mcp.example.com/mcp
    headers:
      Authorization: Bearer ${MY_TOOLS_TOKEN}
    allowed_tools: [search, get_weather]
    max_call_depth: 5
    timeout_seconds: 30

See config.example.yaml and config.example.json for the full template with all options.

---

Deployment

Local development

export OPENAI_API_KEY=sk-your-key
export GATEWAY_CONFIG=./config.yaml
make build && ./bin/ferrogw

Docker Compose (with PostgreSQL)

services:
  ferrogw:
    image: ghcr.io/ferro-labs/ai-gateway:v1.0.0
    ports:
      - "8080:8080"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - GATEWAY_CONFIG=/etc/ferrogw/config.yaml
      - CONFIG_STORE_BACKEND=postgres
      - CONFIG_STORE_DSN=postgresql://ferrogw:ferrogw@db:5432/ferrogw?sslmode=disable
      - API_KEY_STORE_BACKEND=postgres
      - API_KEY_STORE_DSN=postgresql://ferrogw:ferrogw@db:5432/ferrogw?sslmode=disable
      - REQUEST_LOG_STORE_BACKEND=postgres
      - REQUEST_LOG_STORE_DSN=postgresql://ferrogw:ferrogw@db:5432/ferrogw?sslmode=disable
    volumes:
      - ./config.yaml:/etc/ferrogw/config.yaml:ro
    depends_on:
      - db

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: ferrogw
      POSTGRES_PASSWORD: ferrogw
      POSTGRES_DB: ferrogw
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

Kubernetes via Helm

helm repo add ferro-labs https://ferro-labs.github.io/helm-charts
helm repo update
helm install ferro-gw ferro-labs/ai-gateway \
  --set env.OPENAI_API_KEY=sk-your-key

Helm charts: github.com/ferro-labs/helm-charts

---

Migrate to Ferro Labs AI Gateway

From LiteLLM

LiteLLM users can migrate in one step. Ferro Labs AI Gateway is OpenAI-compatible — change one line in your code:

Python (before — LiteLLM):

from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

Python (after — Ferro Labs AI Gateway):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Node.js (after — Ferro Labs AI Gateway):

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "your-ferro-api-key",
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello" }],
});

Why migrate from LiteLLM:

• 14x higher throughput at 150 concurrent users (2,447 vs 175 RPS)
• 23x less memory at peak load (47 MB vs 1,124 MB under streaming)
• Single binary — no Python environment, no pip, no virtualenv
• Predictable latency — p99 stays under 65 ms at 150 VU vs LiteLLM's timeouts at the same concurrency

Config migration:

# LiteLLM config.yaml               # Ferro Labs config.yaml
model_list:                          strategy:
  - model_name: gpt-4o                mode: fallback
    litellm_params:
      model: gpt-4o                  targets:
      api_key: sk-...                  - virtual_key: openai
  - model_name: claude-3-5-sonnet     - virtual_key: anthropic
    litellm_params:
      model: claude-3-5-sonnet       aliases:
      api_key: sk-ant-...              fast: gpt-4o
                                       smart: claude-3-5-sonnet-20241022

Provider API keys are set via environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.) — not in the config file.

From Portkey

Portkey users: Ferro Labs AI Gateway uses the standard OpenAI SDK — no custom headers required in self-hosted mode.

Before (Portkey hosted):

from portkey_ai import Portkey

client = Portkey(api_key="portkey-key")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

After (Ferro Labs AI Gateway self-hosted):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Why migrate from Portkey:

• Fully open source — no per-request pricing, no log limits
• Self-hosted — your data never leaves your infrastructure
• No vendor lock-in — Apache 2.0 license
• MCP support — Portkey self-hosted lacks native MCP
• FerroCloud (coming soon) for teams that want a managed service

From OpenAI SDK directly

No gateway yet? Add Ferro Labs AI Gateway in front of your existing code with a single base_url change. No other code changes required.

# Before — calling OpenAI directly
client = OpenAI(api_key="sk-...")

# After — routing through Ferro Labs AI Gateway
# Gains: failover, caching, rate limiting, cost tracking
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-ferro-api-key",
)

Ferro Labs AI Gateway handles provider failover automatically — if OpenAI is down, your requests fall through to Anthropic or Gemini with zero application code changes.

---

FerroCloud

FerroCloud — the managed version of Ferro Labs AI Gateway with multi-tenancy, analytics, and cost governance — is coming soon.

👉 Join the waitlist at ferrolabs.ai

---

OpenAI SDK Migration

Point existing OpenAI SDK clients to Ferro Labs AI Gateway by changing only the base URL.

Python:

from openai import OpenAI

client = OpenAI(
    api_key="sk-ferro-...",
    base_url="http://localhost:8080/v1",
)

TypeScript:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "sk-ferro-...",
  baseURL: "http://localhost:8080/v1",
});

---

Contributing

We welcome contributions. New providers go in this OSS repo only — never in FerroCloud. See CONTRIBUTING.md for branch strategy, commit conventions, and PR guidelines.

---

Community

• GitHub Discussions
• Discord
• Built with Ferro Labs AI Gateway? Open a PR to add to our showcase.

---

License

Apache 2.0 — see LICENSE.

Documentation

Overview

Package aigateway provides a high-performance, zero-dependency AI gateway for routing requests to large language model (LLM) providers.

The Gateway type is the main entry point: create one with New, register providers with RegisterProvider, load plugins from config with LoadPlugins, and route requests with Route or RouteStream.

Plugins and routing strategies (single, fallback, load-balance, conditional, content-based, ab-test) are configured via Config which can be loaded from a YAML or JSON file using LoadConfig.

Index

• Constants
• func ValidateConfig(cfg Config) error
• type ABVariantConfig
• type CircuitBreakerConfig
• type Condition
• type Config
  • func LoadConfig(path string) (*Config, error)
• type ContentCondition
• type EventHookFunc
• type Gateway
  • func New(cfg Config) (*Gateway, error)
  • func (g *Gateway) AddHook(fn EventHookFunc)
  • func (g *Gateway) AllModels() []providers.ModelInfo
  • func (g *Gateway) Catalog() models.Catalog
  • func (g *Gateway) Close() error
  • func (g *Gateway) Embed(ctx context.Context, req providers.EmbeddingRequest) (*providers.EmbeddingResponse, error)
  • func (g *Gateway) FindByModel(model string) (providers.Provider, bool)
  • func (g *Gateway) FindStreamingByModel(model string) (providers.StreamProvider, bool)
  • func (g *Gateway) GenerateImage(ctx context.Context, req providers.ImageRequest) (*providers.ImageResponse, error)
  • func (g *Gateway) Get(name string) (providers.Provider, bool)
  • func (g *Gateway) GetConfig() Config
  • func (g *Gateway) GetProvider(name string) (providers.Provider, bool)
  • func (g *Gateway) List() []string
  • func (g *Gateway) ListProviders() []string
  • func (g *Gateway) LoadPlugins() error
  • func (g *Gateway) MCPInitDone() <-chan struct{}
  • func (g *Gateway) RegisterPlugin(stage plugin.Stage, p plugin.Plugin) error
  • func (g *Gateway) RegisterProvider(p providers.Provider)
  • func (g *Gateway) ReloadConfig(cfg Config) error
  • func (g *Gateway) Route(ctx context.Context, req providers.Request) (*providers.Response, error)
  • func (g *Gateway) RouteStream(ctx context.Context, req providers.Request) (<-chan providers.StreamChunk, error)
  • func (g *Gateway) StartDiscovery(ctx context.Context, interval time.Duration) error
• type PluginConfig
• type RetryConfig
• type StrategyConfig
• type StrategyMode
• type Target

Constants

const (
	SubjectRequestCompleted = "gateway.request.completed"
	SubjectRequestFailed    = "gateway.request.failed"
)

Event subject constants used when invoking gateway hooks.

Functions

func ValidateConfig

func ValidateConfig(cfg Config) error

ValidateConfig validates a Config for correctness.

Types

type ABVariantConfig

type ABVariantConfig struct {
	// TargetKey is the virtual_key of the provider for this variant.
	TargetKey string `json:"target_key" yaml:"target_key"`
	// Weight is the relative traffic share for this variant.
	// All weights are summed; each variant's fraction is Weight/Total.
	// Zero is treated as 1 (equal distribution).
	Weight float64 `json:"weight" yaml:"weight"`
	// Label is a short human-readable identifier (e.g. "control", "challenger").
	// It is logged with every routed request for observability.
	Label string `json:"label" yaml:"label"`
}

ABVariantConfig defines a single traffic variant for the "ab-test" strategy.

type CircuitBreakerConfig

type CircuitBreakerConfig struct {
	// FailureThreshold is the number of consecutive failures before the circuit
	// opens. Defaults to 5.
	FailureThreshold int `json:"failure_threshold" yaml:"failure_threshold"`
	// SuccessThreshold is the number of consecutive successes in half-open state
	// required to close the circuit. Defaults to 1.
	SuccessThreshold int `json:"success_threshold" yaml:"success_threshold"`
	// Timeout is the duration the circuit stays open before transitioning to
	// half-open (e.g. "30s"). Defaults to "30s".
	Timeout string `json:"timeout" yaml:"timeout"`
}

CircuitBreakerConfig configures the per-provider circuit breaker.

type Condition

type Condition struct {
	Key       string `json:"key" yaml:"key"`
	Value     string `json:"value" yaml:"value"`
	TargetKey string `json:"target_key" yaml:"target_key"`
}

Condition represents a condition for conditional routing.

type Config

type Config struct {
	// Strategy defines how requests are routed (e.g., single, fallback, loadbalance).
	Strategy StrategyConfig `json:"strategy" yaml:"strategy"`
	// Targets is a list of provider targets to route requests to.
	Targets []Target `json:"targets" yaml:"targets"`
	// Plugins configuration (optional).
	Plugins []PluginConfig `json:"plugins,omitempty" yaml:"plugins,omitempty"`
	// Aliases maps friendly model names (e.g. "fast", "smart") to concrete model IDs.
	// Aliases are resolved before routing — they must not reference other aliases.
	Aliases map[string]string `json:"aliases,omitempty" yaml:"aliases,omitempty"`
	// MCPServers configures external MCP tool servers for agentic tool calling.
	// When set, the gateway injects discovered tools into every chat completion
	// request and executes an agentic loop when the LLM returns tool_calls.
	// FerroCloud populates this field from the tenant's mcp_servers table at
	// gateway.New() time — no separate MCPRegistry() public method is exposed.
	MCPServers []mcp.ServerConfig `json:"mcp_servers,omitempty" yaml:"mcp_servers,omitempty"`
	// MCPToolCallAuditFn, if non-nil, is called after every MCP tool invocation.
	// This field cannot be set via JSON or YAML — set it programmatically before
	// calling New. FerroCloud uses it to write async audit entries to the
	// mcp_tool_call_logs table.
	MCPToolCallAuditFn mcp.ToolCallAuditFn `json:"-" yaml:"-"`
}

Config holds the configuration for the AI Gateway.

func LoadConfig

func LoadConfig(path string) (*Config, error)

LoadConfig reads and parses a config file from the given path. Supported formats: JSON (.json), YAML (.yaml, .yml).

type ContentCondition

type ContentCondition struct {
	// Type is the matching rule type.
	Type string `json:"type" yaml:"type"`
	// Value is the substring or regex pattern to match against.
	Value string `json:"value" yaml:"value"`
	// TargetKey is the virtual_key of the provider to route to when this rule matches.
	TargetKey string `json:"target_key" yaml:"target_key"`
}

ContentCondition maps a prompt-content matching rule to a routing target. Used with the "content-based" strategy mode.

Supported types:

• "prompt_contains" — case-insensitive substring match on user messages
• "prompt_not_contains" — true when NO user message contains the value
• "prompt_regex" — Go regular expression match on user messages

type EventHookFunc

type EventHookFunc func(ctx context.Context, subject string, data map[string]interface{})

EventHookFunc is called asynchronously after a gateway event (request completed or failed). It replaces the old EventPublisher interface with a simpler function-based hook pattern.

type Gateway

type Gateway struct {
	// contains filtered or unexported fields
}

Gateway is the main entry point for routing LLM requests.

func New

func New(cfg Config) (*Gateway, error)

New creates a new Gateway instance with the given configuration.

func (*Gateway) AddHook

func (g *Gateway) AddHook(fn EventHookFunc)

AddHook registers an EventHookFunc that is called asynchronously on each completed or failed request. Multiple hooks may be registered; all are invoked for every event on the shared bounded hook worker pool, so hook implementations should return promptly and avoid indefinite blocking.

func (*Gateway) AllModels

func (g *Gateway) AllModels() []providers.ModelInfo

AllModels returns ModelInfo from all registered providers. If auto-discovery has run for a provider, discovered models take precedence over the provider's static model list.

func (*Gateway) Catalog

func (g *Gateway) Catalog() models.Catalog

Catalog returns a shallow copy of the loaded model catalog. A copy is returned so callers cannot mutate the gateway's internal catalog.

func (*Gateway) Close

func (g *Gateway) Close() error

Close cleans up resources.

func (*Gateway) Embed

func (g *Gateway) Embed(ctx context.Context, req providers.EmbeddingRequest) (*providers.EmbeddingResponse, error)

Embed routes an embedding request to the first registered EmbeddingProvider that supports the requested model.

func (*Gateway) FindByModel

func (g *Gateway) FindByModel(model string) (providers.Provider, bool)

FindByModel returns the first registered provider that supports the given model.

func (*Gateway) FindStreamingByModel

func (g *Gateway) FindStreamingByModel(model string) (providers.StreamProvider, bool)

FindStreamingByModel returns the first registered streaming-capable provider that supports the given model.

func (*Gateway) GenerateImage

func (g *Gateway) GenerateImage(ctx context.Context, req providers.ImageRequest) (*providers.ImageResponse, error)

GenerateImage routes an image generation request to the first registered ImageProvider that supports the requested model.

func (*Gateway) Get

func (g *Gateway) Get(name string) (providers.Provider, bool)

Get satisfies providers.ProviderSource (alias for GetProvider).

func (*Gateway) GetConfig

func (g *Gateway) GetConfig() Config

GetConfig returns a copy of the current configuration.

func (*Gateway) GetProvider

func (g *Gateway) GetProvider(name string) (providers.Provider, bool)

GetProvider returns a registered provider by name.

func (*Gateway) List

func (g *Gateway) List() []string

List satisfies providers.ProviderSource (alias for ListProviders).

func (*Gateway) ListProviders

func (g *Gateway) ListProviders() []string

ListProviders returns the names of all registered providers.

func (*Gateway) LoadPlugins

func (g *Gateway) LoadPlugins() error

LoadPlugins initializes and registers plugins from the gateway configuration.

func (*Gateway) MCPInitDone

func (g *Gateway) MCPInitDone() <-chan struct{}

MCPInitDone returns a channel that is closed once all MCP servers have completed their initialization handshake. The channel is pre-closed when no MCP servers are configured.

func (*Gateway) RegisterPlugin

func (g *Gateway) RegisterPlugin(stage plugin.Stage, p plugin.Plugin) error

RegisterPlugin registers a plugin at the given lifecycle stage.

func (*Gateway) RegisterProvider

func (g *Gateway) RegisterProvider(p providers.Provider)

RegisterProvider registers a provider with the gateway.

func (*Gateway) ReloadConfig

func (g *Gateway) ReloadConfig(cfg Config) error

ReloadConfig validates and applies a new configuration, forcing strategy rebuild on next request.

func (*Gateway) Route

func (g *Gateway) Route(ctx context.Context, req providers.Request) (*providers.Response, error)

Route routes a request to the appropriate provider based on the configuration.

func (*Gateway) RouteStream

func (g *Gateway) RouteStream(ctx context.Context, req providers.Request) (<-chan providers.StreamChunk, error)

RouteStream runs before-request plugins then returns a metered streaming response channel. Provider resolution follows the configured strategy mode, then falls back to any registered provider that supports the requested model and streaming. Prometheus metrics and event hooks are emitted when the returned channel drains (matching the behaviour of Route for non-streaming).

When MCP servers are configured the request is routed through Route instead so that the full agentic tool-call loop can run. The final response is wrapped into a single-chunk stream and returned to the caller (Phase 1 behaviour — true final-response streaming is Phase 1.5).

func (*Gateway) StartDiscovery

func (g *Gateway) StartDiscovery(ctx context.Context, interval time.Duration) error

StartDiscovery periodically refreshes model lists from providers that implement DiscoveryProvider. It runs in a background goroutine until ctx is cancelled. interval must be greater than zero; an error is returned otherwise.

type PluginConfig

type PluginConfig struct {
	Name    string                 `json:"name" yaml:"name"`
	Type    string                 `json:"type" yaml:"type"`
	Stage   string                 `json:"stage" yaml:"stage"`
	Enabled bool                   `json:"enabled" yaml:"enabled"`
	Config  map[string]interface{} `json:"config" yaml:"config"`
}

PluginConfig holds plugin configuration.

type RetryConfig

type RetryConfig struct {
	// Attempts is the maximum number of attempts per target (1 = no retries).
	Attempts int `json:"attempts" yaml:"attempts"`
	// OnStatusCodes, when non-empty, limits retries to the listed HTTP status
	// codes. A retry is skipped when the provider returns a code not in the
	// list, and the strategy moves on to the next target immediately.
	// Leave empty to retry on any error (default behaviour).
	// Example: [429, 502, 503]
	OnStatusCodes []int `json:"on_status_codes,omitempty" yaml:"on_status_codes,omitempty"`
	// InitialBackoffMs is the base backoff in milliseconds for the exponential
	// back-off formula: delay = InitialBackoffMs * 2^(attempt-1).
	// Defaults to 100 ms when unset or zero.
	InitialBackoffMs int `json:"initial_backoff_ms,omitempty" yaml:"initial_backoff_ms,omitempty"`
}

RetryConfig defines retry behavior for the fallback strategy.

type StrategyConfig

type StrategyConfig struct {
	Mode       StrategyMode `json:"mode" yaml:"mode"`
	Conditions []Condition  `json:"conditions,omitempty" yaml:"conditions,omitempty"` // For conditional routing
	// ContentConditions defines rules for the content-based routing strategy.
	// Rules are evaluated in order; the first match wins.
	ContentConditions []ContentCondition `json:"content_conditions,omitempty" yaml:"content_conditions,omitempty"`
	// ABVariants defines the weighted variants for the ab-test strategy.
	ABVariants []ABVariantConfig `json:"ab_variants,omitempty" yaml:"ab_variants,omitempty"`
}

StrategyConfig defines the routing strategy.

type StrategyMode

type StrategyMode string

StrategyMode represents the routing strategy mode.

const (
	ModeSingle        StrategyMode = "single"
	ModeFallback      StrategyMode = "fallback"
	ModeLoadBalance   StrategyMode = "loadbalance"
	ModeConditional   StrategyMode = "conditional"
	ModeLatency       StrategyMode = "least-latency"
	ModeCostOptimized StrategyMode = "cost-optimized"
	ModeContentBased  StrategyMode = "content-based"
	ModeABTest        StrategyMode = "ab-test"
)

StrategyMode constants define the supported routing strategies.

type Target

type Target struct {
	// VirtualKey is the unique identifier for the provider (or a virtual key in the vault).
	VirtualKey string `json:"virtual_key" yaml:"virtual_key"`
	// Weight is used for load balancing.
	Weight float64 `json:"weight,omitempty" yaml:"weight,omitempty"`
	// Retry configuration for this target.
	Retry *RetryConfig `json:"retry,omitempty" yaml:"retry,omitempty"`
	// CircuitBreaker configuration for this target (optional).
	CircuitBreaker *CircuitBreakerConfig `json:"circuit_breaker,omitempty" yaml:"circuit_breaker,omitempty"`
}

Target represents a specific provider target.