aigateway

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 26, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

README

Ferro Logo Ferro AI Gateway

The high-performance, open-source control plane for your AI applications.
Route, observe, and secure requests across 100+ LLM providers via a single OpenAI-compatible API.

License Go Go Reference Discord


AI Gateway Architecture

Ferro Gateway is a remarkably fast, lightweight routing tier built in Go. It acts as an intelligent intermediary between your applications and upstream foundation models, effectively transforming fragmented API integration into a unified, secure, and observable infrastructure layer.

Zero SDK changes required. Drop it into your existing OpenAI-reliant code in one line.

✨ Core Capabilities

  • Unified API: Connect to 100+ top-tier models (OpenAI, Anthropic, Gemini, Mistral, Ollama, DeepSeek, and more) using the exact same standard OpenAI request/response format.
  • Smart Routing Engine: Mitigate downtime and optimize costs using 4 robust routing strategies: Single, Fallback (w/ exponential backoff), Weighted Load Balancing, and Conditional (model-based).
  • Transparent Pass-Through Proxy: Automatically forwards requests for non-chat endpoints (like /v1/audio, /v1/images, /v1/files, etc.) directly to the provider. The gateway securely injects your auth credentials while proxying raw bytes!
  • Enterprise Reliability: Native server-sent events (SSE) streaming support and drop-in client compatibility.
  • Extensible Middleware: Intercept requests via pluggable plugins for Guardrails (PII/word filtering), Token Limiting, exact-match Caching, and Request Logging.
  • Secure Access Manager: Centrally issue scoped, auto-expiring API keys with native RBAC. Zero external database required for stand-up.

⚡ Quick Start

Run via Docker

The fastest way to get started is pulling the official image from GitHub Container Registry.

docker run --rm -p 8080:8080 \
  -e OPENAI_API_KEY=sk-your-key \
  ghcr.io/ferro-labs/ai-gateway:latest

# To run the absolute latest unreleased code from main:
# ghcr.io/ferro-labs/ai-gateway:edge

Build from Source

Ensure you have Go 1.24+ installed.

git clone https://github.com/ferro-labs/ai-gateway.git
cd ai-gateway

export OPENAI_API_KEY=sk-your-key
make run
# Server listens locally on :8080

🔌 1-Line Migration

FerroGateway natively speaks the OpenAI spec. Point your existing client SDKs to the Gateway by changing simply the baseURL—no SDK changes, no prompt edits, no refactoring.

Python
from openai import OpenAI

client = OpenAI(
    api_key="sk-ferro-...", # Managed via ferro
    base_url="http://localhost:8080/v1",  # ← Only change this line
)
TypeScript / Node.js
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "sk-ferro-...",
  baseURL: "http://localhost:8080/v1",  // ← Only change this line 
});
cURL
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-ferro-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-3-opus-20240229","messages":[{"role":"user","content":"Hello!"}]}'
  # The gateway automatically detects the model and routes to Anthropic!

🛣️ Project Roadmap

Ferro Gateway is actively developed to support an end-to-end AI operating environment. We are currently transitioning through major foundational and production-grade phases:

  • v0.1.0 — Foundation Release: Core routing, multi-provider execution, basic guardrails, and streaming capabilities.
  • v0.2.0 — Observability & Resilience: Structured telemetry, Prometheus metrics hooks, robust circuit breaking, file-backed key storage, and deep health checks.
  • v0.3.0 — Modality Expansions: Embeddings, Image generation mapping, Cost tracking via pricing tables, and Model aliasing.
  • v0.4.0 — Persistent State: Dedicated Admin API, SQLite/PostgreSQL persistence, robust CRUD configuration portals.
  • v0.5.0 — Advanced Intelligence: Least-latency and Cost-optimized algorithmic routing, A/B Testing modules, and Semantic Caching.
  • v1.0.0 — Production Ready: Helm charts, open-telemetry export, edge caching, and official SDK embeddings.

Review our detailed ROADMAP.md for deeper implementation plans.

🤝 Contributing

We welcome community contributions! The priority areas for ecosystem growth are:

  1. Adding support for new niche LLM providers.
  2. Building new middleware plugins (Guardrails, Modifiers, Analyzers).
  3. Enhancing test coverage and documentation.

Please see our CONTRIBUTING.md for style guidelines and PR processes.

📄 License

FerroGateway is proudly open-source and released under the Apache 2.0 License.

Documentation

Overview

Package aigateway provides a high-performance, zero-dependency AI gateway for routing requests to large language model (LLM) providers.

The Gateway type is the main entry point: create one with New, register providers with RegisterProvider, load plugins from config with LoadPlugins, and route requests with Route or RouteStream.

Plugins and routing strategies (single, fallback, load-balance, conditional) are configured via Config which can be loaded from a YAML or JSON file using LoadConfig.

Index

Constants

View Source 
const (
	SubjectRequestCompleted = "gateway.request.completed"
	SubjectRequestFailed    = "gateway.request.failed"
)

Event subject constants used when publishing gateway lifecycle events.

Variables

This section is empty.

Functions

func ValidateConfig 

func ValidateConfig(cfg Config) error

ValidateConfig validates a Config for correctness.

Types

type Condition 

type Condition struct {
	Key       string `json:"key" yaml:"key"`
	Value     string `json:"value" yaml:"value"`
	TargetKey string `json:"target_key" yaml:"target_key"`
}

Condition represents a condition for conditional routing.

type Config 

type Config struct {
	// Strategy defines how requests are routed (e.g., single, fallback, loadbalance).
	Strategy StrategyConfig `json:"strategy" yaml:"strategy"`
	// Targets is a list of provider targets to route requests to.
	Targets []Target `json:"targets" yaml:"targets"`
	// Plugins configuration (optional).
	Plugins []PluginConfig `json:"plugins,omitempty" yaml:"plugins,omitempty"`
}

Config holds the configuration for the AI Gateway.

func LoadConfig 

func LoadConfig(path string) (*Config, error)

LoadConfig reads and parses a config file from the given path. Supported formats: JSON (.json), YAML (.yaml, .yml).

type EventPublisher 

type EventPublisher interface {
	Publish(ctx context.Context, subject string, payload interface{}) error
}

EventPublisher is satisfied by any event bus implementation. Any type with a Publish method can be used for event publishing.

type Gateway 

type Gateway struct {
	// contains filtered or unexported fields
}

Gateway is the main entry point for routing LLM requests.

func New 

func New(cfg Config) (*Gateway, error)

New creates a new Gateway instance with the given configuration.

func (*Gateway) Close 

func (g *Gateway) Close() error

Close cleans up resources.

func (*Gateway) GetConfig 

func (g *Gateway) GetConfig() Config

GetConfig returns a copy of the current configuration.

func (*Gateway) LoadPlugins 

func (g *Gateway) LoadPlugins() error

LoadPlugins initializes and registers plugins from the gateway configuration.

func (*Gateway) RegisterPlugin 

func (g *Gateway) RegisterPlugin(stage plugin.Stage, p plugin.Plugin) error

RegisterPlugin registers a plugin at the given lifecycle stage.

func (*Gateway) RegisterProvider 

func (g *Gateway) RegisterProvider(p providers.Provider)

RegisterProvider registers a provider with the gateway.

func (*Gateway) ReloadConfig 

func (g *Gateway) ReloadConfig(cfg Config) error

ReloadConfig validates and applies a new configuration, forcing strategy rebuild on next request.

func (*Gateway) Route 

func (g *Gateway) Route(ctx context.Context, req providers.Request) (*providers.Response, error)

Route routes a request to the appropriate provider based on the configuration.

func (*Gateway) RouteStream 

func (g *Gateway) RouteStream(ctx context.Context, req providers.Request) (<-chan providers.StreamChunk, error)

RouteStream runs before-request plugins then returns a streaming response channel. Provider resolution follows the configured strategy mode, then falls back to any registered provider that supports the requested model and streaming.

func (*Gateway) SetEventsPublisher 

func (g *Gateway) SetEventsPublisher(p EventPublisher)

SetEventsPublisher sets the event publisher for the gateway.

type PluginConfig 

type PluginConfig struct {
	Name    string                 `json:"name" yaml:"name"`
	Type    string                 `json:"type" yaml:"type"`
	Stage   string                 `json:"stage" yaml:"stage"`
	Enabled bool                   `json:"enabled" yaml:"enabled"`
	Config  map[string]interface{} `json:"config" yaml:"config"`
}

PluginConfig holds plugin configuration.

type RetryConfig 

type RetryConfig struct {
	Attempts int `json:"attempts" yaml:"attempts"`
}

RetryConfig defines retry behavior.

type StrategyConfig 

type StrategyConfig struct {
	Mode       StrategyMode `json:"mode" yaml:"mode"`
	Conditions []Condition  `json:"conditions,omitempty" yaml:"conditions,omitempty"` // For conditional routing
}

StrategyConfig defines the routing strategy.

type StrategyMode 

type StrategyMode string

StrategyMode represents the routing strategy mode. 

const (
	ModeSingle      StrategyMode = "single"
	ModeFallback    StrategyMode = "fallback"
	ModeLoadBalance StrategyMode = "loadbalance"
	ModeConditional StrategyMode = "conditional"
)

StrategyMode constants define the supported routing strategies.

type Target 

type Target struct {
	// VirtualKey is the unique identifier for the provider (or a virtual key in the vault).
	VirtualKey string `json:"virtual_key" yaml:"virtual_key"`
	// Weight is used for load balancing.
	Weight float64 `json:"weight,omitempty" yaml:"weight,omitempty"`
	// Retry configuration for this target.
	Retry *RetryConfig `json:"retry,omitempty" yaml:"retry,omitempty"`
}

Target represents a specific provider target.

Source Files

View all Source files

Directories

Path Synopsis
cmd
ferrogw command
Package main provides the HTTP handlers for legacy OpenAI completions endpoint.
 Package main provides the HTTP handlers for legacy OpenAI completions endpoint.
ferrogw-cli command
Package main provides the ferrogw-cli command-line tool for managing the FerroGateway.
 Package main provides the ferrogw-cli command-line tool for managing the FerroGateway.
examples
basic command
Package main demonstrates sending a request directly to any configured LLM provider.
 Package main demonstrates sending a request directly to any configured LLM provider.
custom-plugin command
Package main demonstrates how to write and register a custom plugin with FerroGateway.
 Package main demonstrates how to write and register a custom plugin with FerroGateway.
embedded command
Package main demonstrates embedding FerroGateway inside an existing Go HTTP server.
 Package main demonstrates embedding FerroGateway inside an existing Go HTTP server.
fallback command
Package main demonstrates the fallback routing strategy.
 Package main demonstrates the fallback routing strategy.
loadbalance command
Package main demonstrates weighted load balancing across multiple providers.
 Package main demonstrates weighted load balancing across multiple providers.
with-guardrails command
Package main demonstrates using built-in guardrail plugins.
 Package main demonstrates using built-in guardrail plugins.
internal
admin
Package admin provides HTTP handlers for the gateway administration API.
 Package admin provides HTTP handlers for the gateway administration API.
cache
Package cache provides the CacheEntry and Cache interface used by the response-cache plugin.
 Package cache provides the CacheEntry and Cache interface used by the response-cache plugin.
plugins/cache
Package cache provides a response-cache plugin that stores LLM responses in memory and serves them on exact-match cache hits, reducing provider cost and latency for repeated requests.
 Package cache provides a response-cache plugin that stores LLM responses in memory and serves them on exact-match cache hits, reducing provider cost and latency for repeated requests.
plugins/logger
Package logger provides a request-logger plugin that records each LLM request and response to standard output.
 Package logger provides a request-logger plugin that records each LLM request and response to standard output.
plugins/maxtoken
Package maxtoken provides a max-token guardrail plugin that caps the max_tokens and message count on outgoing requests.
 Package maxtoken provides a max-token guardrail plugin that caps the max_tokens and message count on outgoing requests.
plugins/wordfilter
Package wordfilter provides a word-filter guardrail plugin that rejects requests containing blocked words.
 Package wordfilter provides a word-filter guardrail plugin that rejects requests containing blocked words.
strategies
Package strategies implements the routing strategies used by the gateway.
 Package strategies implements the routing strategies used by the gateway.
version
Package version holds build-time version information for FerroGateway binaries.
 Package version holds build-time version information for FerroGateway binaries.
Package plugin defines the Plugin interface and the lifecycle stages used to hook into the gateway request pipeline.
 Package plugin defines the Plugin interface and the lifecycle stages used to hook into the gateway request pipeline.
Package providers defines the Provider interface and shared data types used across all LLM provider implementations.
 Package providers defines the Provider interface and shared data types used across all LLM provider implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.