llmrouter

README

llm-router

A Go library that provides a unified interface for routing requests across multiple LLM providers. Write against one API, deploy across OpenAI, Anthropic, Google Gemini, and any OpenAI-compatible service.

Prerequisites

• Go 1.22+
• API key for at least one supported provider

Installation

go get github.com/bluefunda/llm-router

Quick Start

package main

import (
    "context"
    "fmt"
    "time"

    llmrouter "github.com/bluefunda/llm-router"
    "github.com/bluefunda/llm-router/middleware"
    "github.com/bluefunda/llm-router/providers/openai"
    "github.com/bluefunda/llm-router/providers/anthropic"
)

func main() {
    router := llmrouter.New(
        llmrouter.WithProvider("openai", openai.NewFromEnv("openai", "OPENAI_API_KEY")),
        llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
        llmrouter.WithMiddleware(
            middleware.NewRetryMiddleware(3, time.Second),
            middleware.NewTimeoutMiddleware(60*time.Second),
        ),
    )

    resp, err := router.Complete(context.Background(), &llmrouter.Request{
        Model: "gpt-4o-mini",
        Messages: []llmrouter.Message{
            {Role: llmrouter.RoleUser, Content: "Hello!"},
        },
    })
    if err != nil {
        panic(err)
    }

    fmt.Println(resp.Choices[0].Message.Content)
}

Providers

Each provider is configured via environment variables or explicit options.

Provider	Package	Env Variable	Models
OpenAI	providers/openai	OPENAI_API_KEY	gpt-4o, gpt-4o-mini, gpt-4.1, o4-mini
Anthropic	providers/anthropic	ANTHROPIC_API_KEY	claude-opus-4, claude-sonnet-4, claude-haiku-3.5
Gemini	providers/gemini	GEMINI_API_KEY	gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash-exp
DeepSeek	providers/openai (preset: deepseek)	DEEPSEEK_API_KEY	deepseek-chat, deepseek-coder
Groq	providers/openai (preset: groq)	GROQ_API_KEY	llama-3.3-70b-versatile, mixtral-8x7b
Together	providers/openai (preset: together)	TOGETHER_API_KEY	llama-3.3-70b, mixtral-8x7b
Ollama	providers/openai (preset: ollama)	—	Any locally hosted model

OpenAI-compatible providers

DeepSeek, Groq, Together AI, and Ollama all use the OpenAI provider with a preset name:

openai.NewFromEnv("deepseek", "DEEPSEEK_API_KEY")
openai.NewFromEnv("groq", "GROQ_API_KEY")
openai.NewFromEnv("ollama", "")  // no key needed for local

Gemini

Gemini requires a context.Context during initialization:

geminiProvider, err := gemini.NewFromEnv(ctx)

Configuration

Router options

router := llmrouter.New(
    llmrouter.WithProvider("openai", openaiProvider),
    llmrouter.WithProvider("anthropic", anthropicProvider),
    llmrouter.WithModelMapping("gpt-4o", "openai"),
    llmrouter.WithModelMapping("claude-sonnet-4-20250514", "anthropic"),
    llmrouter.WithFallback("openai", "anthropic"),
    llmrouter.WithMiddleware(retryMw, circuitMw, timeoutMw),
)

Option	Description
WithProvider	Register a named provider
WithModelMapping	Route a model name to a specific provider
WithFallback	Set fallback provider order
WithMiddleware	Attach middleware to the processing chain

Model resolution

The router resolves a model to a provider in this order:

1. Explicit mapping — WithModelMapping("gpt-4o", "openai")
2. Provider name match — model name equals a registered provider name
3. Provider model list — iterates providers and checks Models()

Middleware

Middleware wraps providers with cross-cutting concerns. They are applied in declaration order (first declared = outermost).

Retry

Exponential backoff with configurable max attempts. Non-retryable errors (auth failures, invalid requests, context cancellation) short-circuit immediately.

mw := middleware.NewRetryMiddleware(3, time.Second)
mw.WithMaxDelay(30 * time.Second)

Circuit Breaker

Prevents cascading failures using the sony/gobreaker library. Opens after consecutive failures exceed the threshold and recovers after the timeout period.

mw := middleware.NewCircuitBreakerMiddleware("llm-cb", 5, 30*time.Second)

Timeout

Enforces a deadline on both Complete and Stream calls. Streaming channels emit an EventError on timeout.

mw := middleware.NewTimeoutMiddleware(60 * time.Second)

Streaming

Stream responses arrive as typed events over a channel:

events, err := router.Stream(ctx, &llmrouter.Request{
    Model:    "claude-sonnet-4-20250514",
    Messages: []llmrouter.Message{
        {Role: llmrouter.RoleUser, Content: "Write a haiku about Go."},
    },
})

for event := range events {
    switch event.Type {
    case llmrouter.EventContentDelta:
        fmt.Print(event.Content)
    case llmrouter.EventToolCallDelta:
        // handle tool call
    case llmrouter.EventDone:
        // stream finished
    case llmrouter.EventError:
        log.Fatal(event.Error)
    }
}

Tool Calling

Define tools once and use them across any provider that supports function calling:

tool := llmrouter.Tool{
    Type: "function",
    Function: llmrouter.Function{
        Name:        "get_weather",
        Description: "Get current weather for a location",
        Parameters:  json.RawMessage(`{
            "type": "object",
            "properties": {
                "location": {"type": "string"}
            },
            "required": ["location"]
        }`),
    },
}

resp, _ := router.Complete(ctx, &llmrouter.Request{
    Model:    "gpt-4o-mini",
    Messages: messages,
    Tools:    []llmrouter.Tool{tool},
})

Multimodal

Messages support text, images, and documents via ContentParts:

msg := llmrouter.Message{
    Role: llmrouter.RoleUser,
    ContentParts: []llmrouter.ContentPart{
        {Type: "text", Text: "What's in this image?"},
        {Type: "image_url", ImageURL: &llmrouter.ImageURL{URL: "https://..."}},
    },
}

Error Handling

The library classifies errors for intelligent retry and routing decisions:

Error	Retryable	Description
ErrRateLimited	Yes	Provider rate limit (429)
ErrAuthFailed	No	Invalid API key (401/403)
ErrInvalidRequest	No	Malformed request (400)
ErrCircuitOpen	No	Circuit breaker is open
ErrMaxRetriesExceed	No	All retry attempts exhausted
ErrUnknownModel	No	Model not found in any provider
ErrNoProviders	No	No providers registered

Use llmrouter.IsRetryable(err) and llmrouter.IsRateLimited(err) for programmatic checks.

Project Structure

router.go                      # Core router — provider registry, model resolution, middleware chain
provider.go                    # Provider and Middleware interfaces
types.go                       # Unified request/response types, streaming events, tool definitions
options.go                     # Functional options for router configuration
errors.go                      # Error types and retryability classification
middleware/
  retry.go                     # Retry with exponential backoff
  timeout.go                   # Request timeout enforcement
  circuitbreaker.go            # Circuit breaker (sony/gobreaker)
providers/
  openai/                      # OpenAI + compatible providers (DeepSeek, Groq, Together, Ollama)
  anthropic/                   # Anthropic Claude
  gemini/                      # Google Gemini
examples/
  simple/                      # Basic completion
  streaming/                   # Streaming responses
  tools/                       # Function calling
  fallback/                    # Multi-provider with middleware

License

MIT

Documentation

Index

• Variables
• func IsRateLimited(err error) bool
• func IsRetryable(err error) bool
• type APIError
  • func (e *APIError) Error() string
  • func (e *APIError) Unwrap() error
• type Choice
• type ContentPart
• type Delta
• type Document
• type Event
• type EventType
• type FuncCall
• type FuncRef
• type Function
• type ImageURL
• type Message
• type Middleware
• type Option
  • func WithFallback(providers ...string) Option
  • func WithMiddleware(m ...Middleware) Option
  • func WithModelMapping(model, provider string) Option
  • func WithProvider(name string, p Provider) Option
• type Provider
• type ProviderConfig
• type Request
• type Response
• type Role
• type Router
  • func New(opts ...Option) *Router
  • func (r *Router) AddMiddleware(m Middleware)
  • func (r *Router) Complete(ctx context.Context, req *Request) (*Response, error)
  • func (r *Router) GetProvider(name string) (Provider, bool)
  • func (r *Router) MapModel(model, provider string)
  • func (r *Router) Providers() []string
  • func (r *Router) RegisterProvider(name string, p Provider)
  • func (r *Router) Route(ctx context.Context, req *Request) (<-chan Event, error)
  • func (r *Router) SetFallbacks(providers ...string)
  • func (r *Router) Stream(ctx context.Context, req *Request) (<-chan Event, error)
• type Tool
• type ToolCall
• type ToolChoice
• type Usage

Variables

var (
	ErrUnknownModel     = errors.New("unknown model")
	ErrUnknownProvider  = errors.New("unknown provider")
	ErrNoProviders      = errors.New("no providers registered")
	ErrRateLimited      = errors.New("rate limited")
	ErrContextCanceled  = errors.New("context canceled")
	ErrStreamClosed     = errors.New("stream closed")
	ErrInvalidRequest   = errors.New("invalid request")
	ErrAuthFailed       = errors.New("authentication failed")
	ErrProviderError    = errors.New("provider error")
	ErrCircuitOpen      = errors.New("circuit breaker is open")
	ErrMaxRetriesExceed = errors.New("max retries exceeded")
)

Sentinel errors

Functions

func IsRateLimited

func IsRateLimited(err error) bool

IsRateLimited returns true if the error indicates rate limiting

func IsRetryable

func IsRetryable(err error) bool

IsRetryable returns true if the error is retryable

Types

type APIError

type APIError struct {
	Provider   string
	StatusCode int
	Message    string
	Type       string
	Err        error
}

APIError represents an error from an LLM provider API

func (*APIError) Error

func (e *APIError) Error() string

func (*APIError) Unwrap

func (e *APIError) Unwrap() error

type Choice

type Choice struct {
	Index        int      `json:"index"`
	Message      *Message `json:"message,omitempty"`
	Delta        *Delta   `json:"delta,omitempty"`
	FinishReason string   `json:"finish_reason,omitempty"`
}

Choice represents a completion choice

type ContentPart

type ContentPart struct {
	Type     string    `json:"type"` // "text", "image_url", or "document"
	Text     string    `json:"text,omitempty"`
	ImageURL *ImageURL `json:"image_url,omitempty"`
	Document *Document `json:"document,omitempty"`
}

ContentPart represents a part of a multimodal message

type Delta

type Delta struct {
	Role      Role       `json:"role,omitempty"`
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Delta represents streaming content delta

type Document

type Document struct {
	Base64    string `json:"base64"`
	MediaType string `json:"media_type"` // e.g. "application/pdf"
}

Document represents a document (PDF, etc.) for providers that support it natively

type Event

type Event struct {
	Type     EventType
	Content  string
	Delta    *Delta
	Response *Response
	Error    error
}

Event represents a streaming event

type EventType

type EventType int

EventType represents the type of streaming event

const (
	EventContentDelta  EventType = iota // Text content chunk
	EventToolCallDelta                  // Tool call chunk
	EventDone                           // Stream completed
	EventError                          // Error occurred
)

type FuncCall

type FuncCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

FuncCall represents a function call

type FuncRef

type FuncRef struct {
	Name string `json:"name"`
}

FuncRef references a specific function

type Function

type Function struct {
	Name        string          `json:"name"`
	Description string          `json:"description,omitempty"`
	Parameters  json.RawMessage `json:"parameters,omitempty"`
}

Function represents a function definition

type ImageURL

type ImageURL struct {
	URL       string `json:"url"`
	Detail    string `json:"detail,omitempty"`
	Base64    string `json:"base64,omitempty"`
	MediaType string `json:"media_type,omitempty"`
}

ImageURL represents an image reference with both URL and base64 forms

type Message

type Message struct {
	Role         Role          `json:"role"`
	Content      string        `json:"content"`
	ContentParts []ContentPart `json:"content_parts,omitempty"`
	Name         string        `json:"name,omitempty"`
	ToolCalls    []ToolCall    `json:"tool_calls,omitempty"`
	ToolCallID   string        `json:"tool_call_id,omitempty"`
}

Message represents a chat message

type Middleware

type Middleware interface {
	Wrap(next Provider) Provider
}

Middleware wraps a Provider with additional functionality

type Option

type Option func(*Router)

Option configures the Router

func WithFallback

func WithFallback(providers ...string) Option

WithFallback sets fallback providers in priority order

func WithMiddleware

func WithMiddleware(m ...Middleware) Option

WithMiddleware adds middleware to the processing chain. Use this with middleware from the middleware package:

import "github.com/bluefunda/llm-router/middleware"

router := llmrouter.New(
    llmrouter.WithMiddleware(
        middleware.NewRetryMiddleware(3, time.Second),
        middleware.NewTimeoutMiddleware(60*time.Second),
    ),
)

func WithModelMapping

func WithModelMapping(model, provider string) Option

WithModelMapping maps a model to a specific provider

func WithProvider

func WithProvider(name string, p Provider) Option

WithProvider registers a provider with the router

type Provider

type Provider interface {
	// Name returns the provider identifier (e.g., "openai", "anthropic")
	Name() string

	// Models returns the list of supported model IDs
	Models() []string

	// Complete performs a non-streaming completion
	Complete(ctx context.Context, req *Request) (*Response, error)

	// Stream performs a streaming completion, returning events via channel
	Stream(ctx context.Context, req *Request) (<-chan Event, error)

	// SupportsTools returns whether the provider supports function/tool calling
	SupportsTools() bool
}

Provider is the core interface that all LLM providers must implement

type ProviderConfig

type ProviderConfig struct {
	Name          string
	APIKey        string
	BaseURL       string
	Model         string
	Models        []string
	MaxRetries    int
	Timeout       time.Duration
	CustomHeaders map[string]string // custom HTTP headers (e.g. api-subscription-key)
	// StringContentOnly forces message content to be sent as plain strings
	// instead of structured arrays. Required for some OpenAI-compatible APIs
	// (e.g. Sarvam) that don't support the array content format.
	StringContentOnly bool
}

ProviderConfig holds common configuration for providers

type Request

type Request struct {
	Messages    []Message      `json:"messages"`
	Model       string         `json:"model,omitempty"`
	Tools       []Tool         `json:"tools,omitempty"`
	ToolChoice  *ToolChoice    `json:"tool_choice,omitempty"`
	Temperature *float64       `json:"temperature,omitempty"`
	MaxTokens   *int           `json:"max_tokens,omitempty"`
	TopP        *float64       `json:"top_p,omitempty"`
	Stop        []string       `json:"stop,omitempty"`
	Metadata    map[string]any `json:"metadata,omitempty"`
}

Request represents a unified LLM request

type Response

type Response struct {
	ID       string   `json:"id"`
	Object   string   `json:"object"`
	Created  int64    `json:"created"`
	Model    string   `json:"model"`
	Choices  []Choice `json:"choices"`
	Usage    *Usage   `json:"usage,omitempty"`
	Provider string   `json:"provider"`
}

Response represents a unified LLM response (OpenAI-compatible)

type Role

type Role string

Role represents the message role

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router manages multiple LLM providers and routes requests

func New

func New(opts ...Option) *Router

New creates a new Router with the given options

func (*Router) AddMiddleware

func (r *Router) AddMiddleware(m Middleware)

AddMiddleware adds middleware to the router

func (*Router) Complete

func (r *Router) Complete(ctx context.Context, req *Request) (*Response, error)

Complete performs a non-streaming completion

func (*Router) GetProvider

func (r *Router) GetProvider(name string) (Provider, bool)

GetProvider returns a provider by name

func (*Router) MapModel

func (r *Router) MapModel(model, provider string)

MapModel maps a model name to a specific provider

func (*Router) Providers

func (r *Router) Providers() []string

Providers returns list of registered provider names

func (*Router) RegisterProvider

func (r *Router) RegisterProvider(name string, p Provider)

RegisterProvider adds a provider to the router

func (*Router) Route

func (r *Router) Route(ctx context.Context, req *Request) (<-chan Event, error)

Route sends a request to the appropriate provider and streams the response

func (*Router) SetFallbacks

func (r *Router) SetFallbacks(providers ...string)

SetFallbacks sets the fallback provider order

func (*Router) Stream

func (r *Router) Stream(ctx context.Context, req *Request) (<-chan Event, error)

Stream is an alias for Route for clarity

type Tool

type Tool struct {
	Type     string   `json:"type"`
	Function Function `json:"function"`
}

Tool represents a function/tool definition

type ToolCall

type ToolCall struct {
	ID       string   `json:"id"`
	Type     string   `json:"type"`
	Function FuncCall `json:"function"`
	Index    *int     `json:"index,omitempty"`
}

ToolCall represents a tool invocation

type ToolChoice

type ToolChoice struct {
	Type     string   `json:"type,omitempty"`
	Function *FuncRef `json:"function,omitempty"`
}

ToolChoice controls tool selection

type Usage

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

Usage represents token usage