llmapimux

package module
v0.9.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: MIT Imports: 26 Imported by: 0

README

llmapimux

中文文档

A Go SDK providing http.Handler implementations for proxying and multiplexing LLM API requests across multiple protocols. Translate between OpenAI, Anthropic, and Google Gemini APIs transparently.

Supported Protocols

Protocol Inbound (receive) Outbound (send)
OpenAI Chat Completions Yes Yes
OpenAI Responses API Yes Yes
Anthropic Messages Yes Yes
Gemini GenerateContent Yes Yes

Any inbound protocol can be routed to any outbound protocol — llmapimux handles the conversion automatically via a unified intermediate representation (IR).

Installation

go get github.com/llmapimux/llmapimux

Requires Go 1.21+.

Quick Start

package main

import (
	"context"
	"net/http"

	"github.com/llmapimux/llmapimux"
)

// SimpleRouter routes all requests to a single OpenAI-compatible backend.
type SimpleRouter struct{}

func (r *SimpleRouter) Route(ctx context.Context, info llmapimux.RouteInfo) (llmapimux.RouteResult, error) {
	return llmapimux.RouteResult{
		Protocol: llmapimux.ProtocolOpenAIChat,
		BaseURL:  "https://api.openai.com",
		APIKey:   "sk-your-api-key",
		Model:    info.Model,
	}, nil
}

func main() {
	mux := llmapimux.NewMux(&SimpleRouter{})

	http.Handle("/v1/chat/completions", mux.OpenAIChatHandler())
	http.Handle("/v1/responses", mux.OpenAIResponsesHandler())
	http.Handle("/v1/messages", mux.AnthropicHandler())
	http.Handle("/v1/models/", mux.GeminiHandler()) // trailing slash for prefix matching

	http.ListenAndServe(":8080", nil)
}

This creates a proxy server that accepts requests in any of the 4 protocols and forwards them to an OpenAI backend, converting the protocol as needed.

Architecture

Inbound Request
    │
    ▼
┌─────────────────┐
│  Protocol Handler│ (OpenAI Chat / Responses / Anthropic / Gemini)
│  Decode → IR     │
└────────┬────────┘
         │
    ┌────▼────┐
    │  Router  │ (your routing logic)
    └────┬────┘
         │
┌────────▼────────┐
│  Outbound Client │ (OpenAI Chat / Responses / Anthropic / Gemini)
│  IR → Encode     │
└─────────────────┘
         │
         ▼
   Target LLM API

Key design choices:

  • Unified IR with 2N adapters covers N² protocol paths (4 protocols = 8 adapters for 16 combinations)
  • No same-protocol passthrough — all requests go through decode → IR → encode, ensuring consistent behavior
  • RawExtra side channel preserves protocol-specific fields for same-protocol roundtrips
  • No retry logic — errors are forwarded as-is
  • Context propagation — client disconnect cancels the upstream request
  • Zero external dependencies — only the Go standard library

Core Concepts

Router

The Router interface is the only required component. It decides where each request goes:

type Router interface {
    Route(ctx context.Context, info RouteInfo) (RouteResult, error)
}

RouteInfo provides: RequestID, Model, InboundProtocol, Stream, HasTools, HasMedia, APIKey.

RouteResult specifies: Protocol, BaseURL, APIKey, Model.

Authenticator

Optional inbound authentication:

type Authenticator interface {
    Authenticate(ctx context.Context, apiKey string) error
}

mux := llmapimux.NewMux(router, llmapimux.WithAuthenticator(myAuth))
StatsReporter

Optional observability hook for request lifecycle events:

type StatsReporter interface {
    OnRequestStart(ctx context.Context, e RequestStartEvent)
    OnFirstByte(ctx context.Context, e FirstByteEvent)
    OnStreamChunk(ctx context.Context, e StreamChunkEvent)
    OnComplete(ctx context.Context, e CompleteEvent)
}

mux := llmapimux.NewMux(router, llmapimux.WithStatsReporter(myReporter))

Embed NoopStatsReporter to only implement the methods you need.

Protocol Notes

  • Gemini: model is extracted from URL path (not request body); inbound handler must be registered with trailing-slash path (e.g. /v1/models/) for prefix matching
  • Anthropic: redacted_thinking content round-trips exactly; inbound auth accepts both x-api-key header and Authorization: Bearer token (x-api-key takes precedence)
  • OpenAI Chat: both system and developer roles map to IR system prompt; outbound emits developer role
  • OpenAI Responses: stateless proxy (no previous_response_id support); built-in tools are silently dropped
  • Cross-protocol fields not representable in the target protocol are silently dropped

Streaming

Both streaming and non-streaming modes are supported for all protocol combinations. Streaming responses use Server-Sent Events (SSE). The Stream field in RouteInfo lets your router make protocol-aware decisions.

Testing

go test ./...                          # Unit + integration tests
cd tests/e2e && go test ./...          # E2E tests with real SDK clients

E2E tests use real SDK clients (anthropic-sdk-go, openai-go/v3, google.golang.org/genai) against local fake servers. Real API tests require a .env file and skip automatically when credentials are missing.

License

MIT

Documentation

Index

Constants

View Source 
const FallbackMaxTokens = 16384

FallbackMaxTokens is used when the model is unknown and max_tokens is not specified.

Variables

This section is empty.

Functions

func EncodeAnthropicRequest 

func EncodeAnthropicRequest(req *Request) ([]byte, error)

EncodeAnthropicRequest encodes a unified IR Request into an Anthropic Messages API JSON body.

func EncodeAnthropicResponse 

func EncodeAnthropicResponse(resp *Response) ([]byte, error)

EncodeAnthropicResponse encodes a unified IR Response into an Anthropic Messages API JSON body.

func EncodeAnthropicStreamEvent 

func EncodeAnthropicStreamEvent(event *StreamEvent) (string, []byte, error)

EncodeAnthropicStreamEvent encodes a unified IR StreamEvent into an Anthropic SSE event. Returns the SSE event type string and the JSON data bytes.

func EncodeGeminiRequest 

func EncodeGeminiRequest(req *Request) (model string, body []byte, err error)

EncodeGeminiRequest encodes a unified IR Request into a Gemini GenerateContent API request. Returns the model name (for URL construction) and the JSON body separately.

func EncodeGeminiResponse 

func EncodeGeminiResponse(resp *Response) ([]byte, error)

EncodeGeminiResponse encodes a unified IR Response into a Gemini GenerateContent API JSON body.

func EncodeGeminiStreamChunk 

func EncodeGeminiStreamChunk(event *StreamEvent) ([]byte, error)

EncodeGeminiStreamChunk encodes a unified IR StreamEvent into a Gemini GenerateContentResponse JSON chunk (suitable for an SSE "data:" line).

func EncodeOpenAIChatRequest 

func EncodeOpenAIChatRequest(req *Request) ([]byte, error)

EncodeOpenAIChatRequest encodes a unified IR Request into an OpenAI Chat Completions API JSON body.

func EncodeOpenAIChatResponse 

func EncodeOpenAIChatResponse(resp *Response) ([]byte, error)

EncodeOpenAIChatResponse encodes a unified IR Response into an OpenAI Chat Completions API JSON body.

func EncodeOpenAIChatStreamChunk 

func EncodeOpenAIChatStreamChunk(event *StreamEvent) ([]byte, error)

EncodeOpenAIChatStreamChunk encodes a unified IR StreamEvent into an OpenAI Chat streaming chunk JSON (suitable for an SSE "data:" line).

func EncodeOpenAIResponsesRequest 

func EncodeOpenAIResponsesRequest(req *Request) ([]byte, error)

EncodeOpenAIResponsesRequest encodes a unified IR Request into an OpenAI Responses API JSON body.

func EncodeOpenAIResponsesResponse 

func EncodeOpenAIResponsesResponse(resp *Response) ([]byte, error)

EncodeOpenAIResponsesResponse encodes a unified IR Response into an OpenAI Responses API JSON body.

func EncodeOpenAIResponsesStreamEvent 

func EncodeOpenAIResponsesStreamEvent(event *StreamEvent) (string, []byte, error)

EncodeOpenAIResponsesStreamEvent encodes a unified IR StreamEvent into an OpenAI Responses API SSE event type and JSON data.

func ModelMaxOutputTokens 

func ModelMaxOutputTokens(model string) int

ModelMaxOutputTokens returns the default max output tokens for a given model name. Returns 0 if the model is unknown. Used when encoding to protocols that require max_tokens (e.g., Anthropic) and the inbound request did not specify one.

Types

type AnthropicClient 

type AnthropicClient struct {
	HTTPClient *http.Client // optional, uses http.DefaultClient if nil
}

AnthropicClient sends IR Requests to the Anthropic Messages API.

func (*AnthropicClient) Send 

func (c *AnthropicClient) Send(ctx context.Context, req *Request, cfg OutboundConfig) (*Response, error)

Send encodes an IR Request to Anthropic JSON, sends it, and returns a decoded IR Response.

func (*AnthropicClient) SendStream 

func (c *AnthropicClient) SendStream(ctx context.Context, req *Request, cfg OutboundConfig) (<-chan StreamResult, error)

SendStream encodes an IR Request, sends it with stream=true, and returns a channel of StreamResults.

func (*AnthropicClient) SetHTTPClient added in v0.9.2 

func (c *AnthropicClient) SetHTTPClient(hc *http.Client)

SetHTTPClient allows callers (e.g. Mux WithHTTPClient) to inject a shared *http.Client so connection behavior (proxy, keepalive, pooling) is controlled in one place.

type AttemptAdmission 

type AttemptAdmission struct {
	Permit       AttemptPermit
	WaitDuration time.Duration
	LimitKey     string
	Limit        int
	Active       int
}

AttemptAdmission describes the result of admitting an outbound attempt.

type AttemptController 

type AttemptController interface {
	Acquire(ctx context.Context, info RouteInfo, target RouteResult, routeAttempt int, retryAttempt int) (AttemptAdmission, error)
	RetryDelay(ctx context.Context, info RouteInfo, target RouteResult, sendErr SendError, routeAttempt int, retryAttempt int) (time.Duration, bool)
}

AttemptController can delay, reject, or retry outbound send attempts. routeAttempt is the fallback-chain attempt number, starting at 1. retryAttempt is the retry number for the current target, starting at 0.

type AttemptErrorEvent 

type AttemptErrorEvent struct {
	RequestID    string
	AttemptNum   int         // which route/fallback attempt failed (1 = primary from Route())
	RetryAttempt int         // physical retry attempt within this route target (0 = first try)
	Target       RouteResult // the target that failed
	SendErr      SendError   // error details
	WillRetry    bool
	RetryDelay   time.Duration
}

AttemptErrorEvent is fired each time a send attempt fails and is retried.

type AttemptPermit 

type AttemptPermit interface {
	Release()
}

AttemptPermit is held for the lifetime of one outbound send attempt. For streaming attempts, Release is called only after the stream is fully consumed or the client request is canceled.

type Authenticator 

type Authenticator interface {
	Authenticate(ctx context.Context, apiKey string) error
}

Authenticator validates inbound API keys.

type CBOption 

type CBOption func(*cbConfig)

CBOption configures a CircuitBreakerRouter.

func WithCircuitKeyFunc 

func WithCircuitKeyFunc(fn func(RouteResult) string) CBOption

WithCircuitKeyFunc sets the function used to derive the circuit-breaker map key from a RouteResult. Default is rr.BaseURL.

func WithFailureThreshold 

func WithFailureThreshold(n int) CBOption

WithFailureThreshold sets the number of consecutive failures before the circuit opens. Default is 5.

func WithHalfOpenMax 

func WithHalfOpenMax(n int) CBOption

WithHalfOpenMax sets the maximum number of concurrent probing requests allowed while the circuit is half-open. Default is 1.

func WithOnStateChange 

func WithOnStateChange(fn func(key string, from, to CircuitState)) CBOption

WithOnStateChange registers a callback that fires on circuit state transitions. The callback is called with the circuit's mutex held — it must not block or call back into the CircuitBreakerRouter.

func WithRecoveryTimeout 

func WithRecoveryTimeout(d time.Duration) CBOption

WithRecoveryTimeout sets the duration a circuit stays open before transitioning to half-open. Default is 30s.

func WithShouldTrip 

func WithShouldTrip(fn func(SendError) bool) CBOption

WithShouldTrip sets the function that decides whether a SendError should count as a circuit-breaker failure. By default, 5xx status codes, timeouts, and connection errors trip the circuit; 4xx errors do not.

func WithSuccessThreshold 

func WithSuccessThreshold(n int) CBOption

WithSuccessThreshold sets the number of consecutive successes in half-open state required to close the circuit. Default is 2.

type CandidateFunc 

type CandidateFunc func(info RouteInfo) []RouteResult

CandidateFunc returns an ordered list of candidate targets for a request.

type CircuitBreakerRouter 

type CircuitBreakerRouter struct {
	// contains filtered or unexported fields
}

CircuitBreakerRouter implements the Router interface with circuit breaker logic wrapping a CandidateFunc.

func (*CircuitBreakerRouter) OnError 

func (r *CircuitBreakerRouter) OnError(_ context.Context, info RouteInfo, target RouteResult, sendErr SendError) (RouteResult, error)

OnError handles a failed send attempt: updates circuit state, then tries to find the next healthy candidate.

func (*CircuitBreakerRouter) OnSuccess 

func (r *CircuitBreakerRouter) OnSuccess(_ context.Context, info RouteInfo, target RouteResult)

OnSuccess handles a successful send: resets failure counters and may close a half-open circuit.

func (*CircuitBreakerRouter) Route 

func (r *CircuitBreakerRouter) Route(_ context.Context, info RouteInfo) (RouteResult, error)

Route picks the first healthy candidate from the candidate list.

type CircuitState 

type CircuitState int

CircuitState represents the state of a circuit breaker. 

const (
	CircuitClosed   CircuitState = iota // Normal — requests pass through
	CircuitOpen                         // Tripped — requests rejected
	CircuitHalfOpen                     // Probing — limited requests allowed
)

type Citation 

type Citation struct {
	Kind     CitationKind `json:"kind,omitempty"`
	Title    string       `json:"title,omitempty"`
	URL      string       `json:"url,omitempty"`
	Start    *int         `json:"start,omitempty"`
	End      *int         `json:"end,omitempty"`
	SourceID string       `json:"source_id,omitempty"`
}

Citation represents a citation or annotation attached to a content part.

type CitationKind 

type CitationKind = string

CitationKind identifies the type of citation or annotation. 

const (
	CitationKindCharLocation    CitationKind = "char_location"
	CitationKindWebSearchResult CitationKind = "web_search_result"
	CitationKindURLCitation     CitationKind = "url_citation"
	CitationKindGemini          CitationKind = "gemini_citation"
)

type Client 

type Client interface {
	Send(ctx context.Context, req *Request, cfg OutboundConfig) (*Response, error)
	SendStream(ctx context.Context, req *Request, cfg OutboundConfig) (<-chan StreamResult, error)
}

Client sends an IR Request to a provider and returns an IR Response.

func NewClient 

func NewClient(protocol Protocol) Client

NewClient returns the appropriate outbound client for the given protocol.

type CompleteEvent 

type CompleteEvent struct {
	RequestID        string
	Time             time.Time
	Status           CompletionStatus
	Error            error
	InboundProtocol  Protocol
	OutboundProtocol Protocol

	TTFB         *time.Duration // nil for non-streaming (no TTFT concept)
	TotalLatency time.Duration

	Usage Usage

	OutputThroughput float64
	TPOT             *time.Duration // Time Per Output Token; nil for non-streaming
	Chunks           int            // Total streaming chunks received (0 if non-streaming)

	StopReason  StopReason
	ActualModel string

	IRResponse *Response

	AttemptNum    int // which route/fallback attempt succeeded (1 = no fallback)
	RetryAttempts int // physical retries across this request
	QueueWait     time.Duration
}

type CompletionStatus 

type CompletionStatus string
const (
	CompletionStatusSuccess  CompletionStatus = "success"
	CompletionStatusError    CompletionStatus = "error"
	CompletionStatusCanceled CompletionStatus = "canceled"
)

type ContentPart 

type ContentPart struct {
	Type                ContentType                 `json:"type"`
	Text                *TextContent                `json:"text,omitempty"`
	Image               *ImageContent               `json:"image,omitempty"`
	ToolUse             *ToolUseContent             `json:"tool_use,omitempty"`
	ToolResult          *ToolResultContent          `json:"tool_result,omitempty"`
	ServerToolUse       *ServerToolUseContent       `json:"server_tool_use,omitempty"`
	WebSearchToolResult *WebSearchToolResultContent `json:"web_search_tool_result,omitempty"`
	Video               *VideoContent               `json:"video,omitempty"`
	Document            *DocumentContent            `json:"document,omitempty"`
	Thinking            *ThinkingContent            `json:"thinking,omitempty"`
	RedactedThinking    *RedactedThinkingContent    `json:"redacted_thinking,omitempty"`
	Refusal             *RefusalContent             `json:"refusal,omitempty"`
	Citations           []Citation                  `json:"citations,omitempty"`
	// SourceType records the original ContentType before cross-protocol conversion.
	// Used by downstream consumers (Stats, logging) to identify degradation.
	// Silently dropped on wire — not serialized to any protocol.
	SourceType ContentType `json:"source_type,omitempty"`
	// BlockExtra carries block-level protocol fields that the IR does not model,
	// keyed by their original wire name (e.g. "cache_control"). It exists so that
	// Anthropic prompt-caching breakpoints survive a decode/encode round-trip:
	// dropping cache_control disables caching for the entire request.
	//
	// Only re-emitted when the outbound protocol matches the protocol that
	// produced it, since these keys are not portable across protocols.
	BlockExtra map[string]json.RawMessage `json:"-"`
}

ContentPart is a union type representing a single piece of content in a message.

type ContentType 

type ContentType string

ContentType represents the type of a content part. 

const (
	ContentTypeText                ContentType = "text"
	ContentTypeImage               ContentType = "image"
	ContentTypeToolUse             ContentType = "tool_use"
	ContentTypeToolResult          ContentType = "tool_result"
	ContentTypeServerToolUse       ContentType = "server_tool_use"
	ContentTypeWebSearchToolResult ContentType = "web_search_tool_result"
	ContentTypeVideo               ContentType = "video"
	ContentTypeDocument            ContentType = "document"
	ContentTypeThinking            ContentType = "thinking"
	ContentTypeRedactedThinking    ContentType = "redacted_thinking"
	ContentTypeRefusal             ContentType = "refusal"
)

type DocumentContent 

type DocumentContent struct {
	Data      []byte `json:"data,omitempty"`
	URL       string `json:"url,omitempty"`
	MediaType string `json:"media_type,omitempty"`
	Title     string `json:"title,omitempty"`
}

DocumentContent holds document data or a URL reference.

type FirstByteEvent 

type FirstByteEvent struct {
	RequestID string
	Time      time.Time
	TTFB      time.Duration
}

type GeminiClient 

type GeminiClient struct {
	HTTPClient *http.Client // optional, uses http.DefaultClient if nil
}

GeminiClient sends IR Requests to the Gemini GenerateContent API.

func (*GeminiClient) Send 

func (c *GeminiClient) Send(ctx context.Context, req *Request, cfg OutboundConfig) (*Response, error)

Send encodes an IR Request to Gemini JSON, sends it, and returns a decoded IR Response.

func (*GeminiClient) SendStream 

func (c *GeminiClient) SendStream(ctx context.Context, req *Request, cfg OutboundConfig) (<-chan StreamResult, error)

SendStream encodes an IR Request, sends it to the streaming endpoint, and returns a channel of StreamResults.

func (*GeminiClient) SetHTTPClient added in v0.9.2 

func (c *GeminiClient) SetHTTPClient(hc *http.Client)

SetHTTPClient allows callers (e.g. Mux WithHTTPClient) to inject a shared *http.Client so connection behavior (proxy, keepalive, pooling) is controlled in one place.

type Handler 

type Handler struct {
	// contains filtered or unexported fields
}

Handler is a unified http.Handler that delegates protocol-specific behavior to an inboundCodec and routing decisions to a Router.

func (*Handler) ServeHTTP 

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler.

type ImageContent 

type ImageContent struct {
	Data      []byte `json:"data,omitempty"`
	URL       string `json:"url,omitempty"`
	MediaType string `json:"media_type,omitempty"`
	Detail    string `json:"detail,omitempty"`
}

ImageContent holds image data or a URL reference.

type IncompleteDetails 

type IncompleteDetails struct {
	Reason string `json:"reason,omitempty"`
}

IncompleteDetails holds information about why a response was incomplete.

type Message 

type Message struct {
	Role    Role          `json:"role"`
	Content []ContentPart `json:"content"`
}

Message is a single turn in a conversation.

type Mux 

type Mux struct {
	// contains filtered or unexported fields
}

Mux is the core entry point that creates inbound handlers for a given router.

func NewMux 

func NewMux(router Router, opts ...MuxOption) *Mux

NewMux creates a new Mux with a Router and optional configuration.

func (*Mux) AnthropicHandler 

func (m *Mux) AnthropicHandler() http.Handler

AnthropicHandler returns an http.Handler for Anthropic Messages inbound requests.

func (*Mux) GeminiHandler 

func (m *Mux) GeminiHandler() http.Handler

GeminiHandler returns an http.Handler for Gemini GenerateContent inbound requests.

func (*Mux) OpenAIChatHandler 

func (m *Mux) OpenAIChatHandler() http.Handler

OpenAIChatHandler returns an http.Handler for OpenAI Chat Completions inbound requests.

func (*Mux) OpenAIResponsesHandler 

func (m *Mux) OpenAIResponsesHandler() http.Handler

OpenAIResponsesHandler returns an http.Handler for OpenAI Responses API inbound requests.

type MuxOption 

type MuxOption func(*Mux)

MuxOption configures a Mux.

func WithAttemptController 

func WithAttemptController(controller AttemptController) MuxOption

WithAttemptController sets a controller that can gate and retry physical outbound send attempts. Nil keeps the default no-controller behavior.

func WithAuthenticator 

func WithAuthenticator(auth Authenticator) MuxOption

WithAuthenticator sets an Authenticator on the Mux.

func WithHTTPClient added in v0.9.2 

func WithHTTPClient(c *http.Client) MuxOption

WithHTTPClient sets a shared *http.Client that is injected into every outbound client created by the Mux. This lets callers control connection behavior (proxy, TCP keepalive, timeouts, connection pooling) in one place. When a RouteResult carries a ProxyURL, the injected transport is cloned and only its Proxy field is overridden, so caller dial/keepalive settings are preserved. Nil keeps the default client behavior.

func WithMapDeveloperToSystem added in v0.7.0 

func WithMapDeveloperToSystem(enabled bool) MuxOption

WithMapDeveloperToSystem configures the gateway to emit SystemPrompt as "system" role (instead of "developer") in outbound OpenAI Chat Completions requests. Many downstream OpenAI-compatible providers (e.g. vLLM, some OpenAI-compatible servers) don't support the "developer" message role and will reject requests containing it with a 400 error.

When enabled, the IR's SystemPrompt — which already consolidates all system and developer content (equivalent to vLLM's _consolidate_system_messages) — is emitted as a single "system" role message at position 0 instead of a "developer" role message. This matches the behavior of vLLM PR #43590 ("Fold developer-role input messages into system instructions") adapted to the gateway's IR-based architecture.

func WithPreserveOriginalModel added in v0.5.0 

func WithPreserveOriginalModel(enabled bool) MuxOption

func WithRequestModifier 

func WithRequestModifier(fn RequestModifier) MuxOption

WithRequestModifier sets a RequestModifier that is called before each outbound send attempt, allowing callers to set Request.OutboundExtra.

func WithStatsReporter 

func WithStatsReporter(r StatsReporter) MuxOption

WithStatsReporter sets a StatsReporter on the Mux.

type NoopStatsReporter 

type NoopStatsReporter struct{}

NoopStatsReporter provides empty implementations of all StatsReporter methods. Embed this in your implementation to override only the methods you care about. Also used as the default when no StatsReporter is configured (avoids nil checks).

func (NoopStatsReporter) OnAttemptError 

func (NoopStatsReporter) OnAttemptError(context.Context, AttemptErrorEvent)

func (NoopStatsReporter) OnComplete 

func (NoopStatsReporter) OnComplete(context.Context, CompleteEvent)

func (NoopStatsReporter) OnFirstByte 

func (NoopStatsReporter) OnFirstByte(context.Context, FirstByteEvent)

func (NoopStatsReporter) OnRequestStart 

func (NoopStatsReporter) OnRequestStart(context.Context, RequestStartEvent)

func (NoopStatsReporter) OnStreamChunk 

func (NoopStatsReporter) OnStreamChunk(context.Context, StreamChunkEvent)

type OpenAIChatClient 

type OpenAIChatClient struct {
	HTTPClient *http.Client // optional, uses http.DefaultClient if nil
}

OpenAIChatClient sends IR Requests to the OpenAI Chat Completions API.

func (*OpenAIChatClient) Send 

func (c *OpenAIChatClient) Send(ctx context.Context, req *Request, cfg OutboundConfig) (*Response, error)

Send encodes an IR Request to OpenAI Chat JSON, sends it, and returns a decoded IR Response.

func (*OpenAIChatClient) SendStream 

func (c *OpenAIChatClient) SendStream(ctx context.Context, req *Request, cfg OutboundConfig) (<-chan StreamResult, error)

SendStream encodes an IR Request, sends it with stream=true, and returns a channel of StreamResults.

func (*OpenAIChatClient) SetHTTPClient added in v0.9.2 

func (c *OpenAIChatClient) SetHTTPClient(hc *http.Client)

SetHTTPClient allows callers (e.g. Mux WithHTTPClient) to inject a shared *http.Client so connection behavior (proxy, keepalive, pooling) is controlled in one place.

type OpenAIResponsesClient 

type OpenAIResponsesClient struct {
	HTTPClient *http.Client // optional, uses http.DefaultClient if nil
}

OpenAIResponsesClient sends IR Requests to the OpenAI Responses API.

func (*OpenAIResponsesClient) Send 

func (c *OpenAIResponsesClient) Send(ctx context.Context, req *Request, cfg OutboundConfig) (*Response, error)

Send encodes an IR Request to OpenAI Responses JSON, sends it, and returns a decoded IR Response.

func (*OpenAIResponsesClient) SendStream 

func (c *OpenAIResponsesClient) SendStream(ctx context.Context, req *Request, cfg OutboundConfig) (<-chan StreamResult, error)

SendStream encodes an IR Request, sends it with stream=true, and returns a channel of StreamResults.

func (*OpenAIResponsesClient) SetHTTPClient added in v0.9.2 

func (c *OpenAIResponsesClient) SetHTTPClient(hc *http.Client)

SetHTTPClient allows callers (e.g. Mux WithHTTPClient) to inject a shared *http.Client so connection behavior (proxy, keepalive, pooling) is controlled in one place.

type OutboundConfig 

type OutboundConfig struct {
	BaseURL  string
	APIKey   string
	ProxyURL string      // optional HTTP/HTTPS proxy URL
	Header   http.Header // optional extra headers to send to upstream
}

OutboundConfig holds the base URL and API key for outbound calls. Protocol is determined by the Client implementation.

type Protocol 

type Protocol string
const (
	ProtocolOpenAIChat      Protocol = "openai_chat"
	ProtocolOpenAIResponses Protocol = "openai_responses"
	ProtocolAnthropic       Protocol = "anthropic"
	ProtocolGemini          Protocol = "gemini"
)

type ProviderExtensions 

type ProviderExtensions map[string]json.RawMessage

ProviderExtensions holds provider-specific extension fields as raw JSON values, keyed by a vendor-namespaced string (e.g. "anthropic/thinking"). These are preserved during same-provider round-trips and silently dropped on cross-provider conversion.

type RedactedThinkingContent 

type RedactedThinkingContent struct {
	Data string `json:"data"`
}

RedactedThinkingContent holds redacted thinking data that must round-trip exactly.

type RefusalContent 

type RefusalContent struct {
	Refusal string `json:"refusal"`
}

RefusalContent holds refusal text from the model.

type Request 

type Request struct {
	OriginalModel  string          `json:"-"`
	Model          string          `json:"model"`
	Messages       []Message       `json:"messages"`
	SystemPrompt   []ContentPart   `json:"system_prompt,omitempty"`
	Tools          []Tool          `json:"tools,omitempty"`
	ToolChoice     *ToolChoice     `json:"tool_choice,omitempty"`
	MaxTokens      int             `json:"max_tokens,omitempty"`
	Temperature    *float64        `json:"temperature,omitempty"`
	TopP           *float64        `json:"top_p,omitempty"`
	TopK           *int            `json:"top_k,omitempty"`
	StopSequences  []string        `json:"stop_sequences,omitempty"`
	Stream         bool            `json:"stream,omitempty"`
	Thinking       *ThinkingConfig `json:"thinking,omitempty"`
	ResponseFormat *ResponseFormat `json:"response_format,omitempty"`
	// ProviderExtensions holds provider-specific extension fields.
	// Keys should be vendor-namespaced (e.g. "anthropic/thinking").
	// Silently dropped on cross-provider conversion.
	ProviderExtensions ProviderExtensions `json:"provider_extensions,omitempty"`
	// Protocol-specific fields preserved for same-protocol roundtrip.
	// Managed internally by the library — do not modify.
	RawExtra map[string]json.RawMessage `json:"-"`
	// Extra fields to merge into the outbound request body.
	// Set by RequestModifier before each send attempt.
	OutboundExtra   map[string]json.RawMessage `json:"-"`
	InboundProtocol Protocol                   `json:"-"`

	// MapDeveloperToSystem controls whether the gateway maps the "developer"
	// role to "system" in outbound OpenAI Chat Completions requests.
	//
	// Many downstream OpenAI-compatible providers (e.g. vLLM, some
	// OpenAI-compatible servers) don't support the "developer" message role
	// and will reject requests containing it with a 400 error. When true,
	// SystemPrompt is emitted as a "system" role message instead of
	// "developer". The IR already consolidates all system and developer
	// content into SystemPrompt (equivalent to vLLM's
	// _consolidate_system_messages), so the output is always a single system
	// message at position 0 regardless of how many system/developer messages
	// appeared in the original request.
	//
	// This field is set by the Handler from the Mux option and is not
	// serialized on the wire.
	MapDeveloperToSystem bool `json:"-"`
}

Request is the unified intermediate representation of an LLM API request.

func DecodeAnthropicRequest 

func DecodeAnthropicRequest(body []byte) (*Request, error)

DecodeAnthropicRequest decodes an Anthropic Messages API JSON request body into the unified IR Request type.

func DecodeGeminiRequest 

func DecodeGeminiRequest(urlPath string, body []byte) (*Request, error)

DecodeGeminiRequest decodes a Gemini GenerateContent API request into the unified IR Request type. The model is extracted from the URL path, not the request body.

func DecodeOpenAIChatRequest 

func DecodeOpenAIChatRequest(body []byte) (*Request, error)

DecodeOpenAIChatRequest decodes an OpenAI Chat Completions API JSON request body into the unified IR Request type.

func DecodeOpenAIResponsesRequest 

func DecodeOpenAIResponsesRequest(body []byte) (*Request, error)

DecodeOpenAIResponsesRequest decodes an OpenAI Responses API JSON request body into the unified IR Request type.

type RequestModifier 

type RequestModifier func(ctx context.Context, req *Request, target RouteResult)

RequestModifier is called before each outbound send attempt. It may inspect the IR Request and current RouteResult, and set req.OutboundExtra to inject additional fields into the outbound request body.

type RequestStartEvent 

type RequestStartEvent struct {
	RequestID        string
	Time             time.Time
	InboundProtocol  Protocol
	OutboundProtocol Protocol
	Streaming        bool
	IRRequest        *Request
}

type Response 

type Response struct {
	ID                 string             `json:"id,omitempty"`
	Model              string             `json:"model,omitempty"`
	Created            int64              `json:"created,omitempty"`
	Content            []ContentPart      `json:"content,omitempty"`
	StopReason         StopReason         `json:"stop_reason,omitempty"`
	StopSequence       string             `json:"stop_sequence,omitempty"`
	Usage              Usage              `json:"usage"`
	ProviderExtensions ProviderExtensions `json:"provider_extensions,omitempty"`
}

Response is the unified intermediate representation of an LLM API response.

func DecodeAnthropicResponse 

func DecodeAnthropicResponse(body []byte) (*Response, error)

DecodeAnthropicResponse decodes an Anthropic Messages API JSON response body into the unified IR Response type.

func DecodeGeminiResponse 

func DecodeGeminiResponse(body []byte) (*Response, error)

DecodeGeminiResponse decodes a Gemini GenerateContent API JSON response body into the unified IR Response type.

func DecodeOpenAIChatResponse 

func DecodeOpenAIChatResponse(body []byte) (*Response, error)

DecodeOpenAIChatResponse decodes an OpenAI Chat Completions API JSON response body into the unified IR Response type.

func DecodeOpenAIResponsesResponse 

func DecodeOpenAIResponsesResponse(body []byte) (*Response, error)

DecodeOpenAIResponsesResponse decodes an OpenAI Responses API JSON response body into the unified IR Response type.

type ResponseFormat 

type ResponseFormat struct {
	Type string `json:"type"`
	// Name is the schema name. OpenAI requires response_format.json_schema.name
	// to be present, so it must survive a round-trip; other protocols ignore it.
	Name       string          `json:"name,omitempty"`
	JSONSchema json.RawMessage `json:"json_schema,omitempty"`
	// Strict enables strict schema adherence (OpenAI json_schema.strict).
	Strict bool `json:"strict,omitempty"`
}

ResponseFormat controls the output format of the model.

type Role 

type Role string

Role represents the role of a message sender. 

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type RouteInfo 

type RouteInfo struct {
	RequestID       string
	Model           string
	InboundProtocol Protocol
	Stream          bool
	HasTools        bool
	HasMedia        bool
	APIKey          string
}

RouteInfo carries explicit, immutable routing inputs.

type RouteResult 

type RouteResult struct {
	Protocol Protocol
	BaseURL  string
	APIKey   string
	Model    string
	ProxyURL string      // optional HTTP/HTTPS proxy URL (e.g. "http://proxy:8080")
	Header   http.Header // optional extra headers to send to upstream
}

RouteResult is the outbound target decided by the Router.

type Router 

type Router interface {
	Route(ctx context.Context, info RouteInfo) (RouteResult, error)
	OnError(ctx context.Context, info RouteInfo, target RouteResult, sendErr SendError) (RouteResult, error)
	OnSuccess(ctx context.Context, info RouteInfo, target RouteResult)
}

Router determines the outbound target for each request.

func NewCircuitBreakerRouter 

func NewCircuitBreakerRouter(fn CandidateFunc, opts ...CBOption) Router

NewCircuitBreakerRouter creates a Router with circuit breaker logic. fn provides the ordered candidate list for each request. Options configure thresholds, timeouts, and callbacks.

func RouterFunc 

func RouterFunc(fn func(ctx context.Context, info RouteInfo) (RouteResult, error)) Router

RouterFunc wraps a Route function into a full Router implementation. OnError returns the error directly (no fallback). OnSuccess is a no-op.

type SSEReader 

type SSEReader struct {
	// contains filtered or unexported fields
}

SSEReader reads Server-Sent Events from an io.Reader.

func NewSSEReader 

func NewSSEReader(r io.Reader) *SSEReader

NewSSEReader creates a new SSEReader wrapping the given io.Reader.

func (*SSEReader) LastEventType 

func (r *SSEReader) LastEventType() string

LastEventType returns the event: field value from the most recently read event.

func (*SSEReader) Read 

func (r *SSEReader) Read() ([]byte, error)

Read reads lines until an empty line delimiter, concatenating all data: fields. It returns the assembled data payload. Returns io.EOF when the stream ends.

type SSEWriter 

type SSEWriter struct {
	// contains filtered or unexported fields
}

SSEWriter writes Server-Sent Events to an io.Writer.

func NewSSEWriter 

func NewSSEWriter(w io.Writer) *SSEWriter

NewSSEWriter creates a new SSEWriter wrapping the given io.Writer.

func (*SSEWriter) WriteData 

func (w *SSEWriter) WriteData(data []byte) error

WriteData writes a data-only SSE event and flushes if possible.

func (*SSEWriter) WriteDone 

func (w *SSEWriter) WriteDone() error

WriteDone writes the [DONE] sentinel event and flushes if possible.

func (*SSEWriter) WriteEvent 

func (w *SSEWriter) WriteEvent(event string, data []byte) error

WriteEvent writes an SSE event with both event: and data: fields, then flushes if possible.

type SendError 

type SendError struct {
	AttemptNum  int
	StatusCode  int
	Header      http.Header
	IsTimeout   bool
	IsConnError bool
	Err         error
}

SendError carries structured information about a failed send attempt.

type ServerToolUseContent 

type ServerToolUseContent struct {
	ID        string          `json:"id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments,omitempty"`
}

ServerToolUseContent represents a built-in server tool call, such as Anthropic web_search.

type StatsReporter 

type StatsReporter interface {
	OnRequestStart(ctx context.Context, e RequestStartEvent)
	OnFirstByte(ctx context.Context, e FirstByteEvent)
	OnStreamChunk(ctx context.Context, e StreamChunkEvent)
	OnComplete(ctx context.Context, e CompleteEvent)
	OnAttemptError(ctx context.Context, e AttemptErrorEvent)
}

StatsReporter receives observability events from the unified Handler. Callbacks are serialized per request. Non-streaming callbacks run on the request goroutine; streaming callbacks may run on a dedicated wrapper goroutine. Implementations should be non-blocking or use buffered channels internally. Reporter panics are treated as caller bugs and are NOT recovered by llmapimux.

type StopReason 

type StopReason string

StopReason represents the reason a model stopped generating. 

const (
	StopReasonEndTurn       StopReason = "end_turn"
	StopReasonMaxTokens     StopReason = "max_tokens"
	StopReasonToolUse       StopReason = "tool_use"
	StopReasonStopSequence  StopReason = "stop_sequence"
	StopReasonContentFilter StopReason = "content_filter"
	StopReasonPauseTurn     StopReason = "pause_turn"
)

type StreamChunkEvent 

type StreamChunkEvent struct {
	RequestID       string
	Time            time.Time
	SequenceNum     int
	ElapsedTime     time.Duration
	InterChunkDelay time.Duration
	IREvent         *StreamEvent
}

type StreamError 

type StreamError struct {
	Type    string `json:"type,omitempty"`
	Code    string `json:"code,omitempty"`
	Message string `json:"message,omitempty"`
	Param   string `json:"param,omitempty"`
}

StreamError holds error information from a streaming error event.

type StreamEvent 

type StreamEvent struct {
	Type       StreamEventType `json:"type"`
	Response   *Response       `json:"response,omitempty"`
	Index      int             `json:"index"`
	Delta      *ContentPart    `json:"delta,omitempty"`
	StopReason *StopReason     `json:"stop_reason,omitempty"`
	// StopSequence carries the matched stop sequence when StopReason is
	// StopReasonStopSequence. Protocols without an equivalent drop it.
	StopSequence      string             `json:"stop_sequence,omitempty"`
	Usage             *Usage             `json:"usage,omitempty"`
	Error             *StreamError       `json:"error,omitempty"`
	IncompleteDetails *IncompleteDetails `json:"incomplete_details,omitempty"`
}

StreamEvent is a single event in a streaming response.

func DecodeAnthropicStreamEvent 

func DecodeAnthropicStreamEvent(eventType string, data []byte) (*StreamEvent, error)

DecodeAnthropicStreamEvent decodes an Anthropic SSE event into the unified IR StreamEvent. eventType is from the SSE "event:" line, data is from the SSE "data:" line. Returns nil, nil for ping events (should be skipped by the caller).

func DecodeGeminiStreamChunk 

func DecodeGeminiStreamChunk(data []byte) ([]*StreamEvent, error)

DecodeGeminiStreamChunk decodes a Gemini streaming chunk JSON (the data from an SSE "data:" line) into the unified IR StreamEvent. Gemini sends complete GenerateContentResponse objects per chunk.

func DecodeOpenAIChatStreamChunk deprecated 

func DecodeOpenAIChatStreamChunk(data []byte) (*StreamEvent, error)

DecodeOpenAIChatStreamChunk decodes an OpenAI Chat streaming chunk JSON (the data from an SSE "data:" line) into a single unified IR StreamEvent.

Deprecated: a single OpenAI Chat chunk can legitimately carry several IR events (e.g. content plus finish_reason, or several parallel tool_calls). Prefer DecodeOpenAIChatStreamChunks, which returns all of them. This wrapper returns only the first event and is kept for backwards compatibility.

func DecodeOpenAIChatStreamChunks added in v0.4.0 

func DecodeOpenAIChatStreamChunks(data []byte) ([]*StreamEvent, error)

DecodeOpenAIChatStreamChunks decodes an OpenAI Chat streaming chunk JSON (the data from an SSE "data:" line) into zero or more unified IR StreamEvents.

A single chunk may fan out into several IR events:

  • Providers such as vLLM, DeepSeek and Azure emit content and finish_reason in the same chunk; both the content delta and the stop event must be produced or the final token is lost.
  • Parallel tool calls may be batched into one delta.tool_calls array; each entry becomes its own IR delta keyed by its own index.

func DecodeOpenAIResponsesStreamEvent 

func DecodeOpenAIResponsesStreamEvent(eventType string, data []byte) ([]*StreamEvent, error)

DecodeOpenAIResponsesStreamEvent decodes an OpenAI Responses API SSE event into zero or more unified IR StreamEvents. Returns ([]*StreamEvent, nil) on success (the slice may be nil for events that should be skipped), or (nil, err) on a protocol-level decode failure. A single OpenAI event may fan out into several IR events — for example, a completed web_search_call produces both the server_tool_use and the web_search_tool_result blocks.

eventType is from the SSE "event:" line, data is the JSON from the SSE "data:" line.

type StreamEventType 

type StreamEventType string

StreamEventType identifies the kind of streaming event. 

const (
	StreamEventStart             StreamEventType = "start"
	StreamEventDelta             StreamEventType = "delta"
	StreamEventContentBlockStart StreamEventType = "content_block_start"
	StreamEventContentBlockStop  StreamEventType = "content_block_stop"
	StreamEventStop              StreamEventType = "stop"
	StreamEventError             StreamEventType = "error"
)

type StreamResult 

type StreamResult struct {
	Event *StreamEvent
	Err   error
}

StreamResult carries either a StreamEvent or an error from mid-stream failures.

type TextContent 

type TextContent struct {
	Text string `json:"text"`
}

TextContent holds plain text.

type ThinkingConfig 

type ThinkingConfig struct {
	Mode            string `json:"mode,omitempty"`
	BudgetTokens    int    `json:"budget_tokens,omitempty"`
	Effort          string `json:"effort,omitempty"`
	IncludeThoughts *bool  `json:"include_thoughts,omitempty"`
	Level           string `json:"level,omitempty"`
}

ThinkingConfig controls extended thinking behavior.

type ThinkingContent 

type ThinkingContent struct {
	Thinking  string `json:"thinking"`
	Signature string `json:"signature,omitempty"`
}

ThinkingContent holds extended thinking output from the model.

type Tool 

type Tool struct {
	Type        string                     `json:"type,omitempty"`
	Name        string                     `json:"name"`
	Description string                     `json:"description,omitempty"`
	Parameters  json.RawMessage            `json:"parameters,omitempty"`
	Strict      bool                       `json:"strict,omitempty"`
	ExtraFields map[string]json.RawMessage `json:"extra_fields,omitempty"`
}

Tool describes a tool available to the model.

type ToolChoice 

type ToolChoice struct {
	Type               string   `json:"type"`
	ToolName           string   `json:"tool_name,omitempty"`
	AllowedToolNames   []string `json:"allowed_tool_names,omitempty"`
	AllowParallelCalls *bool    `json:"allow_parallel_calls,omitempty"`
}

ToolChoice controls how the model selects tools.

type ToolResultContent 

type ToolResultContent struct {
	ToolUseID string        `json:"tool_use_id"`
	Name      string        `json:"name,omitempty"`
	Content   []ContentPart `json:"content,omitempty"`
	IsError   bool          `json:"is_error,omitempty"`
}

ToolResultContent represents the result of a tool call.

type ToolUseContent 

type ToolUseContent struct {
	ID        string          `json:"id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments,omitempty"`
}

ToolUseContent represents a tool call made by the model.

type UpstreamHTTPError 

type UpstreamHTTPError struct {
	StatusCode int
	Header     http.Header
	Body       []byte
}

UpstreamHTTPError represents an HTTP 4xx/5xx error returned by the upstream provider.

func (*UpstreamHTTPError) Error 

func (e *UpstreamHTTPError) Error() string

type Usage 

type Usage struct {
	// === Input side ===
	PromptTokens           int `json:"prompt_tokens,omitempty"`             // Total input tokens (includes cache hit)
	PromptCacheHitTokens   int `json:"prompt_cache_hit_tokens,omitempty"`   // Cache hit tokens (Anthropic: cache_read_input_tokens, OpenAI: cached_tokens, Gemini: cachedContentTokenCount)
	PromptCacheWriteTokens int `json:"prompt_cache_write_tokens,omitempty"` // Cache write tokens (Anthropic: cache_creation_input_tokens)
	PromptAudioTokens      int `json:"prompt_audio_tokens,omitempty"`       // Input audio tokens (OpenAI: prompt_details.audio_tokens)

	// === Output side ===
	CompletionTokens             int `json:"completion_tokens,omitempty"`              // Total output tokens
	CompletionReasoningTokens    int `json:"completion_reasoning_tokens,omitempty"`    // Reasoning/thinking tokens (OpenAI: reasoning_tokens, Gemini: thoughtsTokenCount)
	CompletionAudioTokens        int `json:"completion_audio_tokens,omitempty"`        // Output audio tokens (OpenAI: completion_details.audio_tokens)
	CompletionAcceptedPrediction int `json:"completion_accepted_prediction,omitempty"` // Accepted prediction tokens (OpenAI: accepted_prediction_tokens)
	CompletionRejectedPrediction int `json:"completion_rejected_prediction,omitempty"` // Rejected prediction tokens (OpenAI: rejected_prediction_tokens)
	ServerToolUseTokens          int `json:"server_tool_use_tokens,omitempty"`         // Server-side tool tokens (Anthropic: server_tool_use_tokens)

	// === Summary ===
	TotalTokens int `json:"total_tokens,omitempty"` // = PromptTokens + CompletionTokens (usually from upstream)
}

Usage tracks token consumption for a request/response pair. Field names use OpenAI-style terminology as the canonical IR representation; each protocol's converter maps to/from its own wire field names.

type VideoContent added in v0.6.0 

type VideoContent struct {
	Data      []byte `json:"data,omitempty"`
	URL       string `json:"url,omitempty"`
	MediaType string `json:"media_type,omitempty"`
}

VideoContent holds video data or a URL reference.

type WebSearchResult 

type WebSearchResult struct {
	Title string `json:"title,omitempty"`
	URL   string `json:"url,omitempty"`
}

WebSearchResult is a single search hit returned by a built-in web search tool.

type WebSearchToolResultContent 

type WebSearchToolResultContent struct {
	ToolUseID string            `json:"tool_use_id"`
	Content   []WebSearchResult `json:"content,omitempty"`
	IsError   bool              `json:"is_error,omitempty"`
	ErrorCode string            `json:"error_code,omitempty"`
}

WebSearchToolResultContent represents the result block for a web search server tool call.

Source Files

View all Source files

Directories

Path Synopsis
protocol
anthropic
gemini
openaichat
openairesponses

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.