aigateway

README

Ferro Labs AI Gateway

Open-source, OpenAI-compatible AI gateway built in Go.
Route, govern, and observe LLM traffic across multiple providers through one API.

---

Ferro Labs AI Gateway is a lightweight control plane that sits between your app and model providers. It exposes OpenAI-style endpoints (/v1/chat/completions, /v1/models, /v1/embeddings, /v1/images/generations) while handling routing, retries, guardrails, logging, admin controls, and provider auth centrally.

Why use it

• OpenAI-compatible API surface for easier migration and standard client support.
• Multi-provider routing with 6 strategy modes: single, fallback, loadbalance, conditional, least-latency, cost-optimized.
• Built-in resilience via per-target retry controls and circuit breakers.
• Built-in governance hooks via plugin lifecycle stages (before_request, after_request, on_error).
• Operational visibility through structured logs, /metrics, deep /health, admin APIs, and dashboard UI.
• Production-friendly storage options for runtime config, API keys, and request logs (memory, sqlite, postgres).

Current highlights

Recent releases include:

• Provider layer refactor into per-provider subpackages and canonical Name* constants.
• Unified provider factory with ProviderConfig and AllProviders() registry entries.
• Dashboard history rendering hardening (no script-driven innerHTML assignment).
• CORS middleware warning when wildcard mode is active (unset/empty CORS_ORIGINS).

See CHANGELOG.md for full release notes.

Quick start

Run with Docker

docker run --rm -p 8080:8080 \
  -e OPENAI_API_KEY=sk-your-key \
  ghcr.io/ferro-labs/ai-gateway:latest

Build from source

git clone https://github.com/ferro-labs/ai-gateway.git
cd ai-gateway

export OPENAI_API_KEY=sk-your-key
make run

Verify gateway is running

curl -s http://localhost:8080/health | jq
curl -s http://localhost:8080/v1/models | jq '.data | length'

Send your first request

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-ferro-or-upstream-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Say hello from Ferro Gateway"}]
  }' | jq

Core endpoints

Public/API routes

• GET /health — deep provider health summary.
• GET /metrics — Prometheus metrics.
• GET /v1/models — OpenAI-style model list, enriched from catalog when available.
• POST /v1/chat/completions — chat (streaming + non-streaming).
• POST /v1/completions — legacy text completions.
• POST /v1/embeddings — embeddings.
• POST /v1/images/generations — image generation.
• GET /dashboard — built-in admin dashboard page.
• GET /v1/* (other) — transparent proxy pass-through to provider.

Admin routes (Bearer auth required)

Read scope (read_only or admin):

• GET /admin/dashboard
• GET /admin/keys, GET /admin/keys/{id}, GET /admin/keys/usage
• GET /admin/providers, GET /admin/health
• GET /admin/config, GET /admin/config/history
• GET /admin/logs, GET /admin/logs/stats

Admin scope (admin only):

• POST /admin/keys, PUT /admin/keys/{id}, DELETE /admin/keys/{id}
• POST /admin/keys/{id}/revoke, POST /admin/keys/{id}/rotate
• POST /admin/config, PUT /admin/config, DELETE /admin/config
• POST /admin/config/rollback/{version}
• DELETE /admin/logs

Routing strategy modes

Set strategy.mode in your config:

• single — fixed provider target.
• fallback — try targets in order, with per-target retry.
• loadbalance — weighted selection across targets.
• conditional — route by request conditions.
• least-latency — pick lowest observed latency provider.
• cost-optimized — pick cheapest compatible provider from model catalog pricing.

Configuration

Use GATEWAY_CONFIG to load YAML/JSON config at startup.

export GATEWAY_CONFIG=./config.yaml

Minimal production-style example:

strategy:
  mode: fallback

targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
      initial_backoff_ms: 100

  - virtual_key: anthropic
    retry:
      attempts: 2

aliases:
  fast: gpt-4o-mini
  smart: claude-3-5-sonnet-20241022

plugins:
  - name: request-logger
    type: logging
    stage: before_request
    enabled: true
    config:
      level: info
      persist: true
      backend: sqlite
      dsn: ferrogw-requests.db

See config.example.yaml and config.example.json for a full template.

Built-in providers

Canonical provider keys used in config (targets[].virtual_key):

Virtual Key	Provider	Enablement Env
ai21	AI21	AI21_API_KEY
anthropic	Anthropic	ANTHROPIC_API_KEY
azure-foundry	Azure Foundry	AZURE_FOUNDRY_API_KEY + AZURE_FOUNDRY_ENDPOINT
azure-openai	Azure OpenAI	AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_DEPLOYMENT
bedrock	AWS Bedrock	AWS_REGION or AWS_ACCESS_KEY_ID
cohere	Cohere	COHERE_API_KEY
deepseek	DeepSeek	DEEPSEEK_API_KEY
fireworks	Fireworks	FIREWORKS_API_KEY
gemini	Google Gemini	GEMINI_API_KEY
groq	Groq	GROQ_API_KEY
hugging-face	Hugging Face	HUGGING_FACE_API_KEY
mistral	Mistral	MISTRAL_API_KEY
ollama	Ollama	OLLAMA_HOST
openai	OpenAI	OPENAI_API_KEY
perplexity	Perplexity	PERPLEXITY_API_KEY
replicate	Replicate	REPLICATE_API_TOKEN
together	Together AI	TOGETHER_API_KEY
vertex-ai	Vertex AI	VERTEX_AI_PROJECT_ID + VERTEX_AI_REGION + (VERTEX_AI_API_KEY or VERTEX_AI_SERVICE_ACCOUNT_JSON)
xai	xAI	XAI_API_KEY

Built-in plugins

Registered plugin set:

• Guardrails: word-filter, max-token, pii-redact, secret-scan, prompt-shield, schema-guard, regex-guard
• Transform: response-cache
• Logging: request-logger

Inspect available plugins with:

make build-cli
./bin/ferrogw-cli plugins

Persistence backends

Configure state backends with environment variables:

Area	Backend Env	DSN Env	Values
Runtime config	CONFIG_STORE_BACKEND	CONFIG_STORE_DSN	memory (default), sqlite, postgres
API keys	API_KEY_STORE_BACKEND	API_KEY_STORE_DSN	memory (default), sqlite, postgres
Request logs	REQUEST_LOG_STORE_BACKEND	REQUEST_LOG_STORE_DSN	sqlite, postgres (unset = disabled)

SQLite local example:

export CONFIG_STORE_BACKEND=sqlite
export CONFIG_STORE_DSN=./ferrogw-config.db

export API_KEY_STORE_BACKEND=sqlite
export API_KEY_STORE_DSN=./ferrogw-keys.db

export REQUEST_LOG_STORE_BACKEND=sqlite
export REQUEST_LOG_STORE_DSN=./ferrogw-requests.db

PostgreSQL example:

export CONFIG_STORE_BACKEND=postgres
export CONFIG_STORE_DSN='postgresql://user:pass@db:5432/ferrogw?sslmode=require'

export API_KEY_STORE_BACKEND=postgres
export API_KEY_STORE_DSN='postgresql://user:pass@db:5432/ferrogw?sslmode=require'

export REQUEST_LOG_STORE_BACKEND=postgres
export REQUEST_LOG_STORE_DSN='postgresql://user:pass@db:5432/ferrogw?sslmode=require'

Security and production notes

• CORS defaults to wildcard when CORS_ORIGINS is unset/empty. Set explicit origins in production.
• Bootstrap keys (ADMIN_BOOTSTRAP_KEY, ADMIN_BOOTSTRAP_READ_ONLY_KEY) are for first-run setup only.
• Prefer TLS-backed Postgres DSNs (sslmode=require or stronger).
• Use scoped admin keys and rotate periodically.

Bootstrap key quick setup:

export ADMIN_BOOTSTRAP_ENABLED=true
export ADMIN_BOOTSTRAP_KEY='change-me-admin'
export ADMIN_BOOTSTRAP_READ_ONLY_KEY='change-me-readonly'

CLI

ferrogw-cli manages and inspects a running gateway:

make build-cli

./bin/ferrogw-cli validate config.example.yaml
./bin/ferrogw-cli plugins
./bin/ferrogw-cli admin providers list --api-key "$FERROGW_API_KEY"
./bin/ferrogw-cli admin config history --api-key "$FERROGW_API_KEY"

Persistent CLI flags:

• --gateway-url (env: FERROGW_URL, default: http://localhost:8080)
• --api-key (env: FERROGW_API_KEY)
• --format (table, json, yaml)

Development

Common Make targets:

make deps
make fmt
make lint
make test
make test-coverage
make test-integration
make bench

Run release validation locally:

make release-check
make release-dry-run

1-line SDK migration

Point existing OpenAI SDK clients to Ferro Gateway by changing only the base URL.

Python

from openai import OpenAI

client = OpenAI(
    api_key="sk-ferro-...",
    base_url="http://localhost:8080/v1",
)

TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "sk-ferro-...",
  baseURL: "http://localhost:8080/v1",
});

Roadmap

The project roadmap is maintained in ROADMAP.md.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md and follow CODE_OF_CONDUCT.md.

License

Licensed under Apache 2.0.

Documentation

Overview

Package aigateway provides a high-performance, zero-dependency AI gateway for routing requests to large language model (LLM) providers.

The Gateway type is the main entry point: create one with New, register providers with RegisterProvider, load plugins from config with LoadPlugins, and route requests with Route or RouteStream.

Plugins and routing strategies (single, fallback, load-balance, conditional) are configured via Config which can be loaded from a YAML or JSON file using LoadConfig.

Index

• Constants
• func ValidateConfig(cfg Config) error
• type CircuitBreakerConfig
• type Condition
• type Config
  • func LoadConfig(path string) (*Config, error)
• type EventHookFunc
• type Gateway
  • func New(cfg Config) (*Gateway, error)
  • func (g *Gateway) AddHook(fn EventHookFunc)
  • func (g *Gateway) AllModels() []providers.ModelInfo
  • func (g *Gateway) Catalog() models.Catalog
  • func (g *Gateway) Close() error
  • func (g *Gateway) Embed(ctx context.Context, req providers.EmbeddingRequest) (*providers.EmbeddingResponse, error)
  • func (g *Gateway) FindByModel(model string) (providers.Provider, bool)
  • func (g *Gateway) GenerateImage(ctx context.Context, req providers.ImageRequest) (*providers.ImageResponse, error)
  • func (g *Gateway) Get(name string) (providers.Provider, bool)
  • func (g *Gateway) GetConfig() Config
  • func (g *Gateway) GetProvider(name string) (providers.Provider, bool)
  • func (g *Gateway) List() []string
  • func (g *Gateway) ListProviders() []string
  • func (g *Gateway) LoadPlugins() error
  • func (g *Gateway) RegisterPlugin(stage plugin.Stage, p plugin.Plugin) error
  • func (g *Gateway) RegisterProvider(p providers.Provider)
  • func (g *Gateway) ReloadConfig(cfg Config) error
  • func (g *Gateway) Route(ctx context.Context, req providers.Request) (*providers.Response, error)
  • func (g *Gateway) RouteStream(ctx context.Context, req providers.Request) (<-chan providers.StreamChunk, error)
  • func (g *Gateway) StartDiscovery(ctx context.Context, interval time.Duration) error
• type PluginConfig
• type RetryConfig
• type StrategyConfig
• type StrategyMode
• type Target

Constants

const (
	SubjectRequestCompleted = "gateway.request.completed"
	SubjectRequestFailed    = "gateway.request.failed"
)

Event subject constants used when invoking gateway hooks.

Functions

func ValidateConfig

func ValidateConfig(cfg Config) error

ValidateConfig validates a Config for correctness.

Types

type CircuitBreakerConfig

type CircuitBreakerConfig struct {
	// FailureThreshold is the number of consecutive failures before the circuit
	// opens. Defaults to 5.
	FailureThreshold int `json:"failure_threshold" yaml:"failure_threshold"`
	// SuccessThreshold is the number of consecutive successes in half-open state
	// required to close the circuit. Defaults to 1.
	SuccessThreshold int `json:"success_threshold" yaml:"success_threshold"`
	// Timeout is the duration the circuit stays open before transitioning to
	// half-open (e.g. "30s"). Defaults to "30s".
	Timeout string `json:"timeout" yaml:"timeout"`
}

CircuitBreakerConfig configures the per-provider circuit breaker.

type Condition

type Condition struct {
	Key       string `json:"key" yaml:"key"`
	Value     string `json:"value" yaml:"value"`
	TargetKey string `json:"target_key" yaml:"target_key"`
}

Condition represents a condition for conditional routing.

type Config

type Config struct {
	// Strategy defines how requests are routed (e.g., single, fallback, loadbalance).
	Strategy StrategyConfig `json:"strategy" yaml:"strategy"`
	// Targets is a list of provider targets to route requests to.
	Targets []Target `json:"targets" yaml:"targets"`
	// Plugins configuration (optional).
	Plugins []PluginConfig `json:"plugins,omitempty" yaml:"plugins,omitempty"`
	// Aliases maps friendly model names (e.g. "fast", "smart") to concrete model IDs.
	// Aliases are resolved before routing — they must not reference other aliases.
	Aliases map[string]string `json:"aliases,omitempty" yaml:"aliases,omitempty"`
}

Config holds the configuration for the AI Gateway.

func LoadConfig

func LoadConfig(path string) (*Config, error)

LoadConfig reads and parses a config file from the given path. Supported formats: JSON (.json), YAML (.yaml, .yml).

type EventHookFunc

type EventHookFunc func(ctx context.Context, subject string, data map[string]interface{})

EventHookFunc is called asynchronously after a gateway event (request completed or failed). It replaces the old EventPublisher interface with a simpler function-based hook pattern.

type Gateway

type Gateway struct {
	// contains filtered or unexported fields
}

Gateway is the main entry point for routing LLM requests.

func New

func New(cfg Config) (*Gateway, error)

New creates a new Gateway instance with the given configuration.

func (*Gateway) AddHook

func (g *Gateway) AddHook(fn EventHookFunc)

AddHook registers an EventHookFunc that is called asynchronously on each completed or failed request. Multiple hooks may be registered; all are invoked for every event.

func (*Gateway) AllModels

func (g *Gateway) AllModels() []providers.ModelInfo

AllModels returns ModelInfo from all registered providers. If auto-discovery has run for a provider, discovered models take precedence over the provider's static model list.

func (*Gateway) Catalog

func (g *Gateway) Catalog() models.Catalog

Catalog returns a shallow copy of the loaded model catalog. A copy is returned so callers cannot mutate the gateway's internal catalog.

func (*Gateway) Close

func (g *Gateway) Close() error

Close cleans up resources.

func (*Gateway) Embed

func (g *Gateway) Embed(ctx context.Context, req providers.EmbeddingRequest) (*providers.EmbeddingResponse, error)

Embed routes an embedding request to the first registered EmbeddingProvider that supports the requested model.

func (*Gateway) FindByModel

func (g *Gateway) FindByModel(model string) (providers.Provider, bool)

FindByModel returns the first registered provider that supports the given model.

func (*Gateway) GenerateImage

func (g *Gateway) GenerateImage(ctx context.Context, req providers.ImageRequest) (*providers.ImageResponse, error)

GenerateImage routes an image generation request to the first registered ImageProvider that supports the requested model.

func (*Gateway) Get

func (g *Gateway) Get(name string) (providers.Provider, bool)

Get satisfies providers.ProviderSource (alias for GetProvider).

func (*Gateway) GetConfig

func (g *Gateway) GetConfig() Config

GetConfig returns a copy of the current configuration.

func (*Gateway) GetProvider

func (g *Gateway) GetProvider(name string) (providers.Provider, bool)

GetProvider returns a registered provider by name.

func (*Gateway) List

func (g *Gateway) List() []string

List satisfies providers.ProviderSource (alias for ListProviders).

func (*Gateway) ListProviders

func (g *Gateway) ListProviders() []string

ListProviders returns the names of all registered providers.

func (*Gateway) LoadPlugins

func (g *Gateway) LoadPlugins() error

LoadPlugins initializes and registers plugins from the gateway configuration.

func (*Gateway) RegisterPlugin

func (g *Gateway) RegisterPlugin(stage plugin.Stage, p plugin.Plugin) error

RegisterPlugin registers a plugin at the given lifecycle stage.

func (*Gateway) RegisterProvider

func (g *Gateway) RegisterProvider(p providers.Provider)

RegisterProvider registers a provider with the gateway.

func (*Gateway) ReloadConfig

func (g *Gateway) ReloadConfig(cfg Config) error

ReloadConfig validates and applies a new configuration, forcing strategy rebuild on next request.

func (*Gateway) Route

func (g *Gateway) Route(ctx context.Context, req providers.Request) (*providers.Response, error)

Route routes a request to the appropriate provider based on the configuration.

func (*Gateway) RouteStream

func (g *Gateway) RouteStream(ctx context.Context, req providers.Request) (<-chan providers.StreamChunk, error)

RouteStream runs before-request plugins then returns a metered streaming response channel. Provider resolution follows the configured strategy mode, then falls back to any registered provider that supports the requested model and streaming. Prometheus metrics and event hooks are emitted when the returned channel drains (matching the behaviour of Route for non-streaming).

func (*Gateway) StartDiscovery

func (g *Gateway) StartDiscovery(ctx context.Context, interval time.Duration) error

StartDiscovery periodically refreshes model lists from providers that implement DiscoveryProvider. It runs in a background goroutine until ctx is cancelled. interval must be greater than zero; an error is returned otherwise.

type PluginConfig

type PluginConfig struct {
	Name    string                 `json:"name" yaml:"name"`
	Type    string                 `json:"type" yaml:"type"`
	Stage   string                 `json:"stage" yaml:"stage"`
	Enabled bool                   `json:"enabled" yaml:"enabled"`
	Config  map[string]interface{} `json:"config" yaml:"config"`
}

PluginConfig holds plugin configuration.

type RetryConfig

type RetryConfig struct {
	// Attempts is the maximum number of attempts per target (1 = no retries).
	Attempts int `json:"attempts" yaml:"attempts"`
	// OnStatusCodes, when non-empty, limits retries to the listed HTTP status
	// codes. A retry is skipped when the provider returns a code not in the
	// list, and the strategy moves on to the next target immediately.
	// Leave empty to retry on any error (default behaviour).
	// Example: [429, 502, 503]
	OnStatusCodes []int `json:"on_status_codes,omitempty" yaml:"on_status_codes,omitempty"`
	// InitialBackoffMs is the base backoff in milliseconds for the exponential
	// back-off formula: delay = InitialBackoffMs * 2^(attempt-1).
	// Defaults to 100 ms when unset or zero.
	InitialBackoffMs int `json:"initial_backoff_ms,omitempty" yaml:"initial_backoff_ms,omitempty"`
}

RetryConfig defines retry behavior for the fallback strategy.

type StrategyConfig

type StrategyConfig struct {
	Mode       StrategyMode `json:"mode" yaml:"mode"`
	Conditions []Condition  `json:"conditions,omitempty" yaml:"conditions,omitempty"` // For conditional routing
}

StrategyConfig defines the routing strategy.

type StrategyMode

type StrategyMode string

StrategyMode represents the routing strategy mode.

const (
	ModeSingle        StrategyMode = "single"
	ModeFallback      StrategyMode = "fallback"
	ModeLoadBalance   StrategyMode = "loadbalance"
	ModeConditional   StrategyMode = "conditional"
	ModeLatency       StrategyMode = "least-latency"
	ModeCostOptimized StrategyMode = "cost-optimized"
)

StrategyMode constants define the supported routing strategies.

type Target

type Target struct {
	// VirtualKey is the unique identifier for the provider (or a virtual key in the vault).
	VirtualKey string `json:"virtual_key" yaml:"virtual_key"`
	// Weight is used for load balancing.
	Weight float64 `json:"weight,omitempty" yaml:"weight,omitempty"`
	// Retry configuration for this target.
	Retry *RetryConfig `json:"retry,omitempty" yaml:"retry,omitempty"`
	// CircuitBreaker configuration for this target (optional).
	CircuitBreaker *CircuitBreakerConfig `json:"circuit_breaker,omitempty" yaml:"circuit_breaker,omitempty"`
}

Target represents a specific provider target.