langrails

README

LangRails

Unified LLM provider interface for Go. One API, many providers.

import "github.com/promptrails/langrails/llm"

provider := llm.MustNew(llm.OpenAI, "sk-...")
resp, _ := provider.Complete(ctx, &langrails.CompletionRequest{
    Model:    "gpt-4o",
    Messages: []langrails.Message{{Role: "user", Content: "Hello!"}},
})
fmt.Println(resp.Content)

Switch providers by changing one constant:

provider := llm.MustNew(llm.Anthropic, "sk-ant-...")  // or Gemini, Ollama, ...

Install

go get github.com/promptrails/langrails

Features

• 13 providers — OpenAI, Anthropic, Gemini, DeepSeek, Groq, Fireworks, xAI, OpenRouter, Together, Mistral, Cohere, Perplexity, Ollama
• Streaming — Channel-based, idiomatic Go
• Tool calling — Unified interface + automatic tool execution loop
• Chain — Sequential multi-step prompt pipelines
• Graph — LangGraph-style stateful workflow engine
• MCP — Model Context Protocol client
• A2A — Agent-to-Agent protocol client + server
• Structured output — JSON schema across all providers
• Vision / Multimodal — Images + text in messages
• Prompt templates — Jinja-style {{ variable }} syntax
• Memory — Conversation history with token limits
• Retry & Fallback — Composable resilience decorators
• Zero dependencies — Only Go standard library

Documentation

Getting Started	Installation, first request, error handling
Providers	All providers, config examples
Parameters	All parameters, provider support matrix
Streaming	Real-time token streaming
Vision / Multimodal	Images + text in messages
Structured Output	JSON schema constrained output
Prompt Templates	Jinja-style variable substitution
Memory	Conversation history management
Tool Calling	Function calling + automatic tool loop
Chain	Sequential prompt pipelines
Graph	Stateful workflows, conditional routing
MCP	Model Context Protocol integration
A2A	Agent-to-Agent protocol client + server
Retry & Fallback	Resilience patterns

Full docs with search: promptrails.github.io/langrails

License

MIT — PromptRails

Part of the PromptRails AI Toolkit

• LangRails — Unified LLM provider
• GuardRails — Content safety scanning
• MemoryRails — Agent memory
• MediaRails — AI media generation
• Go AI Toolkit — Demo app

Documentation

Overview

Package langrails provides a unified interface for interacting with multiple LLM (Large Language Model) providers through a single, consistent API.

It supports 11+ providers including OpenAI, Anthropic, Google Gemini, DeepSeek, Groq, Fireworks, xAI, OpenRouter, Together, Mistral, and Cohere.

Core Interface

All providers implement the Provider interface with two methods:

• Complete: sends a request and returns the full response
• Stream: sends a request and returns a channel of streaming events

Quick Start

import (
	"github.com/promptrails/langrails/llm/openai"
)

provider := openai.New("sk-...")
resp, err := provider.Complete(ctx, &langrails.CompletionRequest{
	Model:    "gpt-4o",
	Messages: []langrails.Message{{Role: "user", Content: "Hello!"}},
})

Streaming

events, err := provider.Stream(ctx, &langrails.CompletionRequest{
	Model:    "gpt-4o",
	Messages: []langrails.Message{{Role: "user", Content: "Hello!"}},
})
for event := range events {
	if event.Type == langrails.EventContent {
		fmt.Print(event.Content)
	}
}

Provider Decorators

Providers can be wrapped with decorators for retry and fallback behavior:

provider := langrails.WithRetry(openai.New("sk-..."), 3)
provider = langrails.WithFallback(provider, anthropic.New("sk-..."))

Index

• type APIError
  • func (e *APIError) Error() string
  • func (e *APIError) IsAuthError() bool
  • func (e *APIError) IsRateLimitError() bool
  • func (e *APIError) IsRetryable() bool
  • func (e *APIError) IsServerError() bool
• type CompletionRequest
• type CompletionResponse
• type ContentPart
  • func ImageBase64Part(data string, mediaType string) ContentPart
  • func ImageURLPart(url string) ContentPart
  • func TextPart(text string) ContentPart
• type EventType
• type FallbackProvider
  • func WithFallback(primary, fallback Provider) *FallbackProvider
  • func (f *FallbackProvider) Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)
  • func (f *FallbackProvider) Stream(ctx context.Context, req *CompletionRequest) (<-chan StreamEvent, error)
• type Message
• type Provider
• type RetryOption
  • func WithBaseDelay(d time.Duration) RetryOption
• type RetryProvider
  • func WithRetry(provider Provider, maxRetries int, opts ...RetryOption) *RetryProvider
  • func (r *RetryProvider) Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)
  • func (r *RetryProvider) Stream(ctx context.Context, req *CompletionRequest) (<-chan StreamEvent, error)
• type StreamEvent
• type TokenUsage
• type ToolCall
• type ToolDefinition

Types

type APIError

type APIError struct {
	// StatusCode is the HTTP status code returned by the provider.
	StatusCode int

	// Message is the human-readable error message.
	Message string

	// Provider is the name of the provider that returned the error.
	Provider string
}

APIError represents an error response from an LLM provider's API.

func (*APIError) Error

func (e *APIError) Error() string

Error implements the error interface.

func (*APIError) IsAuthError

func (e *APIError) IsAuthError() bool

IsAuthError returns true if the error is an authentication/authorization error (401/403).

func (*APIError) IsRateLimitError

func (e *APIError) IsRateLimitError() bool

IsRateLimitError returns true if the error is a rate limit error (429).

func (*APIError) IsRetryable

func (e *APIError) IsRetryable() bool

IsRetryable returns true if the request can be retried. Rate limit errors and server errors are considered retryable.

func (*APIError) IsServerError

func (e *APIError) IsServerError() bool

IsServerError returns true if the error is a server-side error (5xx).

type CompletionRequest

type CompletionRequest struct {
	// Model is the provider-specific model identifier (e.g., "gpt-4o", "claude-sonnet-4-20250514").
	Model string

	// Messages is the conversation history to send to the model.
	Messages []Message

	// SystemPrompt is an optional system instruction. Some providers handle
	// this as a separate field rather than a system message.
	SystemPrompt string

	// Temperature controls randomness in the output. Range is typically 0-2.
	// nil means the provider's default is used.
	Temperature *float64

	// MaxTokens limits the maximum number of tokens in the response.
	// nil means the provider's default is used.
	MaxTokens *int

	// TopP controls nucleus sampling. Range is typically 0-1.
	// nil means the provider's default is used.
	TopP *float64

	// TopK limits the number of tokens considered at each step.
	// Supported by Anthropic and Gemini. Ignored by OpenAI-compatible providers.
	TopK *int

	// FrequencyPenalty penalizes tokens based on their frequency in the output so far.
	// Range is typically -2 to 2. Supported by OpenAI-compatible providers.
	FrequencyPenalty *float64

	// PresencePenalty penalizes tokens based on whether they appear in the output so far.
	// Range is typically -2 to 2. Supported by OpenAI-compatible providers.
	PresencePenalty *float64

	// Stop is a list of sequences where the model should stop generating.
	// The model will stop at the first occurrence of any stop sequence.
	Stop []string

	// Seed enables deterministic output when supported by the provider.
	// Same seed + same request = same output (best effort).
	Seed *int

	// Tools defines the functions/tools available for the model to call.
	// Not all providers support tool calling.
	Tools []ToolDefinition

	// OutputSchema is an optional JSON schema for structured output.
	// When set, the provider will attempt to constrain the output to match
	// this schema. Support varies by provider.
	OutputSchema *[]byte

	// Thinking enables extended thinking / chain-of-thought mode.
	// When true, Anthropic returns thinking blocks, and OpenAI uses
	// reasoning effort for o-series models.
	Thinking bool

	// ThinkingBudget limits the number of thinking tokens (Anthropic only).
	// Ignored when Thinking is false.
	ThinkingBudget *int
}

CompletionRequest represents a request to an LLM provider.

type CompletionResponse

type CompletionResponse struct {
	// Content is the generated text content.
	Content string

	// Thinking contains the model's internal reasoning when Thinking mode
	// is enabled. Only populated by providers that support extended thinking
	// (e.g., Anthropic).
	Thinking string

	// ToolCalls contains any tool/function calls the model wants to make.
	// When non-empty, the caller should execute the tools and send the
	// results back in a follow-up request.
	ToolCalls []ToolCall

	// Usage contains token usage statistics for this request.
	Usage TokenUsage

	// FinishReason indicates why the model stopped generating.
	// Common values: "stop", "tool_calls", "length", "content_filter".
	FinishReason string

	// Model is the actual model that was used (may differ from the requested model).
	Model string
}

CompletionResponse represents the response from an LLM provider.

type ContentPart

type ContentPart struct {
	// Type is the content type: "text" or "image".
	Type string

	// Text is the text content. Only used when Type is "text".
	Text string

	// ImageURL is the URL of the image. Only used when Type is "image".
	// Can be an HTTP(S) URL or a base64 data URI (data:image/png;base64,...).
	ImageURL string
}

ContentPart represents a part of a multimodal message. A message can contain multiple parts, mixing text and images.

func ImageBase64Part

func ImageBase64Part(data string, mediaType string) ContentPart

ImageBase64Part creates an image content part from base64-encoded data. mediaType should be "image/png", "image/jpeg", etc.

func ImageURLPart

func ImageURLPart(url string) ContentPart

ImageURLPart creates an image content part from a URL.

func TextPart

func TextPart(text string) ContentPart

TextPart creates a text content part.

type EventType

type EventType string

EventType represents the type of a streaming event.

const (
	// EventContent indicates a text content chunk.
	EventContent EventType = "content"

	// EventToolCall indicates a tool/function call event.
	EventToolCall EventType = "tool_call"

	// EventDone indicates the stream has completed successfully.
	EventDone EventType = "done"

	// EventError indicates an error occurred during streaming.
	EventError EventType = "error"
)

type FallbackProvider

type FallbackProvider struct {
	// contains filtered or unexported fields
}

FallbackProvider wraps two providers, trying the primary first and falling back to the secondary if the primary returns an error.

Fallback providers can be chained to create a priority list:

provider := langrails.WithFallback(
	openai.New("sk-..."),
	anthropic.New("sk-ant-..."),
)

func WithFallback

func WithFallback(primary, fallback Provider) *FallbackProvider

WithFallback wraps two providers in a fallback chain. If the primary provider returns an error, the fallback provider is tried.

func (*FallbackProvider) Complete

func (f *FallbackProvider) Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)

Complete tries the primary provider first, then falls back to the secondary provider on any error.

func (*FallbackProvider) Stream

func (f *FallbackProvider) Stream(ctx context.Context, req *CompletionRequest) (<-chan StreamEvent, error)

Stream tries the primary provider first, then falls back to the secondary provider on any error.

type Message

type Message struct {
	// Role is the role of the message sender.
	// Valid values: "system", "user", "assistant", "tool".
	Role string

	// Content is the text content of the message.
	// For simple text-only messages, set this field.
	Content string

	// ContentParts is an optional list of content parts for multimodal messages.
	// When set, this takes precedence over Content. Use this to send images
	// alongside text. If nil, Content is used as a single text part.
	ContentParts []ContentPart

	// ToolCallID is the ID of the tool call this message is responding to.
	// Only used when Role is "tool".
	ToolCallID string

	// ToolCalls contains tool/function calls made by the assistant.
	// Only present when Role is "assistant" and the model wants to call tools.
	ToolCalls []ToolCall
}

Message represents a single message in a conversation.

type Provider

type Provider interface {
	// Complete sends a completion request and blocks until the full response
	// is received. It returns the complete response or an error.
	Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)

	// Stream sends a completion request and returns a channel that emits
	// events as they arrive. The channel is closed when the response is
	// complete or an error occurs. Callers should range over the channel
	// and check each event's Type field.
	Stream(ctx context.Context, req *CompletionRequest) (<-chan StreamEvent, error)
}

Provider is the unified interface that all LLM providers must implement. It provides both synchronous and streaming completion methods.

type RetryOption

type RetryOption func(*RetryProvider)

RetryOption configures the retry behavior.

func WithBaseDelay

func WithBaseDelay(d time.Duration) RetryOption

WithBaseDelay sets the base delay for exponential backoff. Default is 1 second. The actual delay doubles with each retry: 1s, 2s, 4s, 8s, etc.

type RetryProvider

type RetryProvider struct {
	// contains filtered or unexported fields
}

RetryProvider wraps a Provider with automatic retry logic using exponential backoff. Only retryable errors (rate limits, server errors) are retried.

func WithRetry

func WithRetry(provider Provider, maxRetries int, opts ...RetryOption) *RetryProvider

WithRetry wraps a provider with retry logic. maxRetries is the maximum number of retry attempts (not including the initial attempt).

Example:

provider := langrails.WithRetry(openai.New("sk-..."), 3)

func (*RetryProvider) Complete

func (r *RetryProvider) Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)

Complete sends a completion request with automatic retries.

func (*RetryProvider) Stream

func (r *RetryProvider) Stream(ctx context.Context, req *CompletionRequest) (<-chan StreamEvent, error)

Stream sends a streaming request with automatic retries. Note: only the initial connection is retried, not mid-stream failures.

type StreamEvent

type StreamEvent struct {
	// Type indicates the kind of event.
	Type EventType

	// Content contains the text chunk for EventContent events.
	Content string

	// ToolCall contains tool call data for EventToolCall events.
	ToolCall *ToolCall

	// Error contains error details for EventError events.
	Error error

	// Usage contains token usage data, typically sent with the final event.
	Usage *TokenUsage
}

StreamEvent represents a single event in a streaming response.

type TokenUsage

type TokenUsage struct {
	// PromptTokens is the number of tokens in the input/prompt.
	PromptTokens int

	// CompletionTokens is the number of tokens in the generated output.
	CompletionTokens int

	// TotalTokens is the sum of PromptTokens and CompletionTokens.
	TotalTokens int
}

TokenUsage contains token consumption statistics for a completion request.

type ToolCall

type ToolCall struct {
	// ID is a unique identifier for this tool call, used to match
	// tool results back to the original call.
	ID string

	// Name is the name of the tool to call.
	Name string

	// Arguments is a JSON-encoded string of the arguments to pass to the tool.
	Arguments string

	// Metadata holds provider-specific data that must be round-tripped through
	// the conversation history. For example, Gemini requires thoughtSignature
	// to be returned with function call parts in subsequent requests.
	Metadata map[string]string
}

ToolCall represents a request from the model to call a specific tool.

type ToolDefinition

type ToolDefinition struct {
	// Name is the unique identifier for this tool.
	Name string

	// Description explains what the tool does, helping the model
	// decide when and how to use it.
	Description string

	// Parameters is a JSON schema describing the tool's input parameters.
	Parameters json.RawMessage
}

ToolDefinition describes a tool/function that the model can call.