llmhub

package module
v0.0.0-...-a455755 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 9 Imported by: 2

README

llmhub

Unified, provider-agnostic Go client for modern Large Language Models (LLMs). llmhub wraps multiple vendors (OpenAI, Anthropic, Gemini, Ollama, OpenRouter, Z.AI, and your own) behind a single, expressive API that understands multi-modal messages, streaming, and provider registries.

Why llmhub?

  • One API, many vendors – swap providers without rewriting your business logic.
  • Multi-modal ready – mix text and images in both requests and responses.
  • Tool calling – declare provider-agnostic tools and handle normalized tool calls.
  • Streaming made simple – consume deltas through idiomatic Go channels.
  • Extensible registry – register first-party or external providers at runtime.
  • Functional options – configure models, endpoints, and credentials cleanly.

Installation

go get github.com/smhanov/llmhub

Quick Start

package main

import (
    "context"
    "fmt"

    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/openai"
)

func main() {
    client, err := llmhub.New("openai", "sk-YOUR-KEY", llmhub.WithModel("gpt-4o-mini"))
    if err != nil {
        panic(err)
    }

    prompt := []*llmhub.Message{
        llmhub.NewSystemMessage(llmhub.Text("You are a witty assistant.")),
        llmhub.NewUserMessage(llmhub.Text("Explain quantum mechanics in five words.")),
    }

    resp, err := client.Generate(context.Background(), prompt)
    if err != nil {
        panic(err)
    }

    fmt.Println(resp.Text())
}

Streaming Responses

stream, err := client.Stream(ctx, prompt)
if err != nil {
    // HTTP 4xx/5xx from the provider, including 429, fail here so callers
    // can fail over before writing downstream streaming headers.
    log.Fatal(err)
}
for chunk := range stream {
    if chunk.Err != nil {
        log.Printf("stream error: %v", chunk.Err)
        break
    }
    if chunk.ReasoningDelta != "" {
        log.Printf("reasoning delta: %s", chunk.ReasoningDelta)
    }
    fmt.Print(chunk.Delta)
    if chunk.Done {
        break
    }
}

Vision & Multi-modal Inputs

prompt := []*llmhub.Message{
    llmhub.NewUserMessage(
        llmhub.Text("What is shown here?"),
        llmhub.Image("https://example.com/diagram.png"),
    ),
}
resp, _ := client.Generate(ctx, prompt)
for _, part := range resp.Content {
    if text, ok := part.(*llmhub.TextContent); ok {
        fmt.Println(text.Text)
    }
}

Reasoning / Thinking Blocks

Some models expose reasoning as separate blocks in the response payload. llmhub preserves those blocks in Response.Content as *llmhub.ReasoningContent.

resp, _ := client.Generate(ctx, prompt)

fmt.Println("final answer:", resp.Text())
fmt.Println("reasoning:", resp.ReasoningText())

for _, part := range resp.Content {
    if r, ok := part.(*llmhub.ReasoningContent); ok {
        fmt.Println("reasoning block:", r.Text)
    }
}

For streaming, reasoning is exposed separately on each chunk via StreamChunk.ReasoningDelta.

Tool Calling

Declare tools with WithTools, read requested calls from Response.ToolCalls(), execute them in your code, then send the result back with NewToolResultMessage.

weather := llmhub.NewTool("weather", "Get current weather", map[string]interface{}{
    "type": "object",
    "properties": map[string]interface{}{
        "city": map[string]interface{}{"type": "string"},
    },
    "required": []string{"city"},
})

client, _ := llmhub.New("openai", apiKey,
    llmhub.WithModel("gpt-4o-mini"),
    llmhub.WithTools(weather),
    llmhub.WithToolChoice(llmhub.AutoToolChoice()),
)

messages := []*llmhub.Message{
    llmhub.NewUserMessage(llmhub.Text("What is the weather in Toronto?")),
}

resp, _ := client.Generate(ctx, messages)
for _, call := range resp.ToolCalls() {
    result := runTool(call.Name, call.Arguments)
    messages = append(messages,
        llmhub.NewAssistantMessage(llmhub.ToolCall(call.ID, call.Name, call.Arguments)),
        llmhub.NewToolResultMessage(call.ID, call.Name, llmhub.Text(result)),
    )
}

finalResp, _ := client.Generate(ctx, messages)
fmt.Println(finalResp.Text())

Tool choice helpers include AutoToolChoice, NoToolChoice, RequiredToolChoice, and NamedToolChoice("tool_name") for providers that expose tool-choice controls. Streaming tool calls are exposed on StreamChunk.ToolCalls. Parallel OpenAI-style argument fragments include ToolCallContent.Index so callers can reassemble concurrent tool calls.

Provider Tool Calling Support
OpenAI ✅ Chat Completions tools
Anthropic ✅ Messages API tools
Gemini ✅ Function declarations
Ollama ✅ Native /api/chat tools
xAI ✅ Chat Completions tools
OpenRouter ✅ Chat Completions tools
Z.AI ✅ Chat Completions tools

Provider Registry

Add your own provider in a different module:

func init() {
    llmhub.MustRegisterProvider("my-llm", func(apiKey string, opts ...llmhub.Option) (llmhub.Provider, error) {
        return newMyClient(apiKey, opts...) // implement llmhub.Provider
    })
}

At runtime, consumers simply call llmhub.New("my-llm", "token").

Built-in Providers

Provider Status Notes
OpenAI ✅ Production Chat Completions, multi-modal prompts, SSE streaming.
Anthropic ✅ Production Claude 3 Messages API with streaming deltas.
Gemini ✅ Production Gemini 1.5 multi-modal text+vision APIs, streaming JSON.
Ollama ✅ Production Local inference via /api/chat, streaming friendly.
xAI ✅ Production Grok models, API key or OAuth device flow, SSE streaming.
OpenRouter ✅ Production OpenAI-compatible gateway to many vendors, returns per-request cost.
Z.AI ✅ Production GLM Chat Completions; exact versioned base URLs, no forced /v1.
OpenAI Provider Details

Automatic /v1 suffix: When a custom base URL is provided (via WithBaseURL), the OpenAI provider appends /v1 unless the path already ends in a version segment (/v1, /v4, …). So https://api.openai.com and https://api.openai.com/v1 behave identically, and a gateway that already pins a version (e.g. https://api.z.ai/api/paas/v4) is left unchanged.

"default" model: When the model is set to "default" (case-insensitive), the provider queries the /v1/models endpoint at initialization and automatically selects the first available model. This is especially useful for self-hosted OpenAI-compatible servers (e.g. Ollama, vLLM, LocalAI) where you may not know the model name in advance:

client, err := llmhub.New("openai", "key",
    llmhub.WithBaseURL("http://localhost:11434"),
    llmhub.WithModel("default"),
)
// The provider will query http://localhost:11434/v1/models and use the first model.
xAI (Grok) Provider Details

Import providers/xai to register the xai provider. It uses the OpenAI-compatible Chat Completions API, supports streaming and tool calling, and defaults to grok-4.6. Override the model when your account has access to a different Grok model.

For xAI API keys, use it exactly like any other API-key provider—OAuth setup is not involved:

import (
    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/xai"
)

// Using an API key:
client, err := llmhub.New("xai", apiKey,
    llmhub.WithModel("grok-4.6"),
)

if err != nil {
    panic(err)
}
OpenRouter Provider Details

Import providers/openrouter to register the openrouter provider. It uses OpenRouter's OpenAI-compatible Chat Completions API (https://openrouter.ai/api/v1), supports streaming and tool calling, and can route to any model available on OpenRouter using its vendor/model identifiers (e.g. x-ai/grok-4.5, openai/gpt-4o-mini):

import (
    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/openrouter"
)

client, err := llmhub.New("openrouter", apiKey,
    llmhub.WithModel("x-ai/grok-4.5"),
)

OpenRouter returns the actual per-request cost in every response (usage.cost), so Response.Usage.Cost is always the exact amount charged— no WithCost rates are needed for OpenRouter.

Z.AI Provider Details

Import providers/zai to register the zai provider. It uses Z.AI's OpenAI-compatible Chat Completions API and keeps the versioned base URL exactly as supplied (no automatic /v1 suffix). New defaults to the general endpoint; NewCodingPlan uses the Coding Plan endpoint.

import (
    "github.com/smhanov/llmhub"
    "github.com/smhanov/llmhub/providers/zai"
)

client, err := llmhub.New("zai", apiKey,
    llmhub.WithBaseURL(zai.CodingPlanBaseURL),
    llmhub.WithModel("glm-5.3"),
)

To reduce binary size, providers self-register when imported, enabling tree-shaking when unused.

import (
    _ "github.com/smhanov/llmhub/providers/openai"
    _ "github.com/smhanov/llmhub/providers/anthropic"
    _ "github.com/smhanov/llmhub/providers/gemini"
    _ "github.com/smhanov/llmhub/providers/ollama"
    _ "github.com/smhanov/llmhub/providers/openrouter"
    _ "github.com/smhanov/llmhub/providers/xai"
    _ "github.com/smhanov/llmhub/providers/zai"
)

Each provider reads the shared functional options:

  • WithAPIKey – supply SaaS credentials (openai, anthropic, gemini, xai, openrouter, zai).
  • WithTokenSource – supply an OAuth / dynamic token source (xai).
  • WithBaseURL – point to proxies/self-hosted gateways.
  • WithModel, WithTemperature – customize LLM behavior per call. Often it is best to omit and go with the defaults.
  • WithMaxTokens – only set this when you truly need a hard output cap; otherwise leave it unset to reduce the risk of truncated responses.
  • WithWebSearch – enable web search/grounding (Gemini: google_search tool).
  • WithTools, WithToolChoice – enable user-defined tool calling.
  • WithResponseModalities – control output modalities (e.g. "IMAGE" for Gemini image generation).
  • WithCost – set per-million-token pricing for cost accounting.
  • WithRetryOnStatus – override HTTP retry for a status (WithRetryOnStatus(429, false) surfaces rate limits immediately).

[!WARNING] Prefer not to use WithMaxTokens in normal application code. Provider defaults usually produce more complete answers, while an explicit cap that is too low commonly causes cut-off output.

Image Generation / Output Modalities

Gemini image-generation models (e.g. gemini-2.5-flash-image) can return images instead of—or alongside—text. Use WithResponseModalities to tell the model which output types you want:

import (
    "github.com/smhanov/llmhub"
    _ "github.com/smhanov/llmhub/providers/gemini"
)

client, _ := llmhub.New("gemini", apiKey,
    llmhub.WithModel("gemini-2.5-flash-image"),
    llmhub.WithResponseModalities("IMAGE"),
)

prompt := []*llmhub.Message{
    llmhub.NewUserMessage(
        llmhub.Text("Upscale this image to 800 pixels wide."),
        llmhub.Image("data:image/jpeg;base64,/9j/4AAQ..."),
    ),
}

resp, _ := client.Generate(ctx, prompt)

for _, part := range resp.Content {
    if img, ok := part.(*llmhub.ImageContent); ok {
        // img.URL is a data URL: "data:image/png;base64,..."
        fmt.Println("Got image:", len(img.URL), "bytes")
    }
}

Pass "TEXT" and "IMAGE" together to allow mixed text+image output:

llmhub.WithResponseModalities("TEXT", "IMAGE")
Provider Image Output Support
Gemini ✅ Via WithResponseModalities("IMAGE")
OpenAI ❌ Use the Images API directly
Anthropic ❌ Not supported
Ollama ❌ Not supported
OpenRouter ❌ Not supported

Cost Accounting

llmhub can track the estimated cost of each request based on token usage and configured per-million-token rates. Costs are expressed in US dollars per 1 million tokens, matching standard LLM provider pricing.

client, _ := llmhub.New("openai", apiKey,
    llmhub.WithModel("gpt-4o"),
    llmhub.WithCost(2.50, 10.00), // $2.50 per 1M input, $10.00 per 1M output tokens
)

resp, _ := client.Generate(ctx, prompt)
fmt.Printf("Tokens: %d in, %d out\n",
    resp.Usage.PromptTokens, resp.Usage.CompletionTokens)
fmt.Printf("Cost: $%.6f\n", resp.Usage.Cost)

Cost is computed automatically after each Generate call:

$$ \text{Cost} = \frac{\text{PromptTokens} \times \text{InputRate}}{1{,}000{,}000}

  • \frac{\text{CompletionTokens} \times \text{OutputRate}}{1{,}000{,}000} $$

If the LLM provider returns cost directly in its API response (e.g., OpenRouter's usage.cost), that cost takes precedence and overrides any rates configured via WithCost. This also applies to OpenAI-compatible gateways and self-hosted servers that report cost or total_cost in the usage block. When no provider cost is returned and no cost rates are configured, Usage.Cost will be zero.

You can also override cost rates on a per-request basis:

// Use cheaper rates for a specific call
resp, _ := client.Generate(ctx, prompt,
    llmhub.WithCost(0.15, 0.60),
)

Web Search / Grounding

Some providers support web search to ground responses in real-time information:

client, _ := llmhub.New("gemini", apiKey,
    llmhub.WithModel("gemini-2.5-flash"),
    llmhub.WithWebSearch(true),
)

prompt := []*llmhub.Message{
    llmhub.NewUserMessage(llmhub.Text("What are the latest news about Apple Inc?")),
}

resp, _ := client.Generate(ctx, prompt)
fmt.Println(resp.Text())
Provider Web Search Support
Gemini ✅ Uses google_search tool
OpenAI ❌ Not supported
Anthropic ❌ Not supported
Ollama ❌ Not supported
xAI ❌ Not supported
OpenRouter ❌ Not supported

OAuth Token Sources

OAuth is opt-in and provider-dependent. It is not required by existing API-key providers. Currently, xai is the built-in provider that accepts WithTokenSource; future OAuth providers can reuse the generic auth and auth/oauth2 packages.

OAuth login is always explicit and application-controlled: llmhub.New, Generate, and Stream will never open a browser or wait for a user to sign in.

Using an Existing Token File
import (
    "github.com/smhanov/llmhub"
    "github.com/smhanov/llmhub/providers/xai"
)

// Discover an existing ~/.grok/auth.json, ~/.xgroxy/auth.json, or XAI_AUTH_FILE.
authPath := xai.DefaultAuthPath("")
store := xai.NewFileTokenStore(authPath)
source := xai.NewTokenSource(store)

client, err := llmhub.New("xai", "",
    llmhub.WithTokenSource(source),
    llmhub.WithModel("grok-4.6"),
)
if err != nil {
    panic(err)
}

WithTokenSource takes precedence over the positional apiKey argument and WithAPIKey. If neither one is supplied to xai, construction fails with llmhub.ErrInvalidInput.

Performing an Interactive Device Login

Login is an explicit, application-controlled operation that never blocks inside library constructors:

import (
    "context"
    "fmt"

    "github.com/smhanov/llmhub/providers/xai"
)

func login(ctx context.Context) error {
    store := xai.NewFileTokenStore(xai.DefaultAuthPath(""))
    flow := xai.NewDeviceFlow()

    authz, err := flow.Start(ctx)
    if err != nil {
        return err
    }

    fmt.Printf("Please visit: %s\n", authz.VerificationURI)
    if authz.VerificationURIComplete != "" {
        fmt.Printf("Or open: %s\n", authz.VerificationURIComplete)
    }
    fmt.Printf("And enter code: %s\n", authz.UserCode)

    token, err := flow.Wait(ctx, authz)
    if err != nil {
        return err
    }

    return store.Save(ctx, token)
}
Credential Precedence & Error Handling

  • Precedence: WithTokenSource takes precedence over WithAPIKey and the positional apiKey argument.

  • Refresh: xai.NewTokenSource refreshes access tokens before they expire and performs one refresh/retry after an API 401 response. It safely handles rotating refresh tokens within one process.

  • Reauthentication: If a refresh token is expired or revoked, provider calls return an error wrapping auth.ErrReauthenticationRequired. Use errors.Is to prompt for a new interactive login:

    if errors.Is(err, auth.ErrReauthenticationRequired) {
    // Start the device flow again and save the new token.
}

  • Security: New token directories/files are created with private 0700/0600 permissions on Unix. Do not print, commit, or pass access/refresh tokens as command-line arguments.

xAI OAuth acceptance example

examples/xai-oauth is a real end-to-end example. It performs an explicit device login when needed, can force a token refresh, then verifies both a non-streaming and a streaming Grok response. It requires an authorized xAI account and intentionally is not part of the offline test suite.

go run ./examples/xai-oauth \
  -auth-file /private/path/auth.json \
  -force-login \
  -verify-refresh \
  -model grok-4.6

The command prints a verification URL and one-time code, waits for you to authorize it in a browser, and succeeds only after it prints:

refresh: OK
generate: OK
stream: OK
xai oauth acceptance: PASS

Use a dedicated private auth-file path for this check. The example never prints access or refresh tokens.

Need multi-provider routing? Instantiate one llmhub.Client per provider and switch at runtime:

openaiClient := llmhub.MustNew("openai", os.Getenv("OPENAI_API_KEY"), llmhub.WithModel("gpt-4o"))
claudeClient := llmhub.MustNew("anthropic", os.Getenv("ANTHROPIC_API_KEY"), llmhub.WithModel("claude-3-opus-20240229"))

func answer(ctx context.Context, prompt []*llmhub.Message, vendor string) (*llmhub.Response, error) {
    switch vendor {
    case "anthropic":
        return claudeClient.Generate(ctx, prompt)
    default:
        return openaiClient.Generate(ctx, prompt)
    }
}

Testing

go test ./...

CLI Test Tool

A command-line tool is included for end-to-end testing of providers. Build and run it from the repository root:

go run ./examples/cli [options]
Options
Flag Description
-provider Provider name: openai, anthropic, gemini, ollama, xai, openrouter (required)
-model Model identifier (e.g., gpt-4o, claude-3-haiku-20240307, gemini-2.5-flash, grok-4.6, x-ai/grok-4.5)
-api-key API key (or use env vars OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, XAI_API_KEY, OPENROUTER_API_KEY)
-auth-file Path to token file for OAuth providers (e.g. xAI)
-base-url Override provider base URL (useful for Ollama or proxies)
-prompt Text prompt to send
-prompt-file File containing the prompt text
-images Comma-separated list of image file paths or URLs
-stream Enable streaming mode
-temperature Sampling temperature (default: 0.7)
-max-tokens Hard cap on generated tokens; leave unset unless needed to avoid truncation
-input-cost Cost per 1M input tokens in USD (for cost accounting)
-output-cost Cost per 1M output tokens in USD (for cost accounting)
-timeout Request timeout duration (e.g. 30s, 2m, 10m)
Examples

Text generation with Ollama (self-hosted):

go run ./examples/cli \
  -provider ollama \
  -model qwen3:32b \
  -base-url https://ollama.example.com \
  -prompt "Why is the sky blue?"

Text generation with Gemini:

go run ./examples/cli \
  -provider gemini \
  -model gemini-2.5-flash \
  -api-key YOUR_GEMINI_KEY \
  -prompt "Explain quantum entanglement simply."

Vision/image input with Gemini:

go run ./examples/cli \
  -provider gemini \
  -model gemini-2.5-flash \
  -api-key YOUR_GEMINI_KEY \
  -prompt "Describe this image in detail." \
  -images cat.jpg

Streaming mode with OpenAI:

go run ./examples/cli \
  -provider openai \
  -model gpt-4o \
  -api-key YOUR_OPENAI_KEY \
  -prompt "Write a haiku about coding." \
  -stream

OpenRouter with provider-reported cost:

go run ./examples/cli \
  -provider openrouter \
  -model x-ai/grok-4.5 \
  -api-key YOUR_OPENROUTER_KEY \
  -prompt "Reply with exactly: OK"
# Tokens and the exact Cost charged by OpenRouter are printed automatically.

Using environment variables:

export OPENAI_API_KEY=sk-...
go run ./examples/cli -provider openai -model gpt-4o -prompt "Hello!"

Using an existing xAI OAuth token file:

go run ./examples/cli \
  -provider xai \
  -auth-file ~/.grok/auth.json \
  -model grok-4.6 \
  -prompt "Reply with exactly: OK"

With cost accounting:

go run ./examples/cli \
  -provider openai \
  -model gpt-4o \
  -input-cost 2.50 \
  -output-cost 10.00 \
  -prompt "Explain Go interfaces."

Contributing

Issues and PRs are welcome! Start by filing an issue describing the provider or feature you would like to add, then open a PR with tests and documentation. Check the existing provider stubs (Anthropic, Gemini, Ollama) for extension points.

This project uses a rebase-only workflow for merging into main: PRs are rebased onto main and merged with a fast-forward rebase (gh pr merge --rebase) — never merge commits or squash. Keep your branches up to date with git rebase main and keep commits small and clean, as they land in main individually.

License

MIT License © 2026 llmhub contributors

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrProviderNotFound is returned when the requested provider has not been registered.
	ErrProviderNotFound = errors.New("llmhub: provider not found")
	// ErrProviderAlreadyRegistered signals that a provider name has already been registered.
	ErrProviderAlreadyRegistered = errors.New("llmhub: provider already registered")
	// ErrNotImplemented is used by provider stubs that have not yet been wired up.
	ErrNotImplemented = errors.New("llmhub: feature not implemented")
	// ErrInvalidInput wraps validation failures on user-supplied data.
	ErrInvalidInput = errors.New("llmhub: invalid input")
)

Functions

func ApplyOptions 

func ApplyOptions(cfg *Config, opts ...Option)

ApplyOptions mutates a Config in-place with the provided options.

func MustRegisterProvider 

func MustRegisterProvider(name string, factory ProviderFactory)

MustRegisterProvider registers a provider and panics on failure.

func RegisterProvider 

func RegisterProvider(name string, factory ProviderFactory) error

RegisterProvider adds a provider factory to the global registry.

func RegisteredProviders 

func RegisteredProviders() []string

RegisteredProviders returns a sorted slice of registered provider names.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the main entry point for interacting with LLM providers via the unified API.

func MustNew 

func MustNew(providerName, apiKey string, opts ...Option) *Client

MustNew is a helper that panics if client creation fails.

func New 

func New(providerName, apiKey string, opts ...Option) (*Client, error)

New creates a Client bound to the named provider.

func Wrap 

func Wrap(provider Provider, opts ...Option) *Client

Wrap binds an already-constructed provider to a Client.

func (*Client) Generate 

func (c *Client) Generate(ctx context.Context, prompt []*Message, opts ...Option) (*Response, error)

Generate performs a single request/response interaction with the provider.

func (*Client) ProviderName 

func (c *Client) ProviderName() string

ProviderName returns the underlying provider identifier.

func (*Client) Stream 

func (c *Client) Stream(ctx context.Context, prompt []*Message, opts ...Option) (<-chan StreamChunk, error)

Stream initiates a streaming interaction and returns a read-only channel of chunks.

type Config 

type Config struct {
	Model       string
	Temperature float64
	// MaxTokens applies a hard cap to generated output tokens.
	// Leave this unset unless you specifically need that cap, because values
	// that are too low can cause the model to return truncated output.
	MaxTokens       int
	APIKey          string
	TokenSource     auth.TokenSource
	BaseURL         string
	HTTPClient      *http.Client
	Headers         map[string]string
	ExtraBody       map[string]json.RawMessage
	EnableWebSearch bool // Enables web search/grounding (Gemini: google_search, Perplexity: always on)
	Tools           []Tool
	ToolChoice      *ToolChoice

	// RetryOnStatus overrides whether HTTP-backed providers retry a given
	// status code. true means retry with backoff; false means return the
	// response immediately. Statuses not present keep the default (retry
	// 429 only). Set via WithRetryOnStatus.
	RetryOnStatus map[int]bool

	// ResponseModalities controls the output modalities the model should
	// produce. For example, setting this to []string{"IMAGE"} tells the
	// Gemini image-generation models to return an image instead of text.
	// Leave nil for the provider default (text).
	ResponseModalities []string

	// Cost accounting: prices expressed per 1 million tokens.
	InputCostPerMillionTokens  float64
	OutputCostPerMillionTokens float64
}

Config captures all tunable request options shared across providers.

func NewConfig 

func NewConfig(opts ...Option) Config

NewConfig produces a Config populated by the provided options.

func (Config) Clone 

func (c Config) Clone() Config

Clone produces a copy of the config suitable for per-request overrides.

type ContentPart 

type ContentPart interface {
	Type() string
}

ContentPart is implemented by any structure that can be included inside a message.

type Error 

type Error struct {
	Provider string
	Op       string
	Err      error
}

Error captures provider-specific failures with additional context.

func (*Error) Error 

func (e *Error) Error() string

Error returns the string form of the wrapped error.

func (*Error) Unwrap 

func (e *Error) Unwrap() error

Unwrap exposes the underlying error for errors.Is/As support.

type ImageContent 

type ImageContent struct {
	URL    string
	Detail string // optional granularity instruction used by some providers
}

ImageContent represents a reference to an image by URL or base64 payload.

func Image 

func Image(url string) *ImageContent

Image is a helper constructor for an image part.

func (*ImageContent) Type 

func (i *ImageContent) Type() string

Type identifies the piece as an image.

type Message 

type Message struct {
	Role    Role
	Content []ContentPart
	Meta    map[string]string
}

Message represents one turn in a conversation with the provider and can mix modalities.

func NewAssistantMessage 

func NewAssistantMessage(parts ...ContentPart) *Message

NewAssistantMessage returns a message authored by the assistant.

func NewMessage 

func NewMessage(role Role, parts ...ContentPart) *Message

NewMessage constructs a message with the provided role and content parts.

func NewSystemMessage 

func NewSystemMessage(parts ...ContentPart) *Message

NewSystemMessage returns a message authored by the system.

func NewToolMessage 

func NewToolMessage(parts ...ContentPart) *Message

NewToolMessage returns a message authored by a tool.

func NewToolResultMessage 

func NewToolResultMessage(toolCallID, name string, parts ...ContentPart) *Message

NewToolResultMessage returns a message containing the result for a tool call.

func NewUserMessage 

func NewUserMessage(parts ...ContentPart) *Message

NewUserMessage returns a message authored by the end-user.

func (*Message) Append 

func (m *Message) Append(parts ...ContentPart)

Append adds one or more content parts to the message.

type Option 

type Option func(*Config)

Option mutates a Config in a functional-options friendly way.

func WithAPIKey 

func WithAPIKey(key string) Option

WithAPIKey stores the credential used by the provider.

func WithBaseURL 

func WithBaseURL(url string) Option

WithBaseURL overrides the provider base URL (useful for proxies and on-prem).

func WithCost 

func WithCost(inputCostPerMillionTokens, outputCostPerMillionTokens float64) Option

WithCost sets the cost per 1 million tokens (input and output) in US dollars. This is used to compute the estimated cost of each request based on token usage.

func WithExtraBody 

func WithExtraBody(extra map[string]json.RawMessage) Option

WithExtraBody adds arbitrary additional fields to the outbound JSON request body. On key collision, these fields override the standard generated fields. Applies only to OpenAI-compatible providers (OpenAI, OpenRouter, xAI); other providers ignore it. Values must be valid JSON.

func WithHTTPClient 

func WithHTTPClient(client *http.Client) Option

WithHTTPClient swaps the HTTP client used by HTTP-backed providers.

func WithHeader 

func WithHeader(key, value string) Option

WithHeader injects a custom header for every request.

func WithMaxTokens 

func WithMaxTokens(max int) Option

WithMaxTokens applies a hard cap to the number of generated tokens.

Prefer leaving this unset unless you specifically need a strict output limit, because setting it too low often causes truncated responses.

func WithModel 

func WithModel(model string) Option

WithModel selects the target model identifier.

func WithResponseModalities 

func WithResponseModalities(modalities ...string) Option

WithResponseModalities specifies the output modalities the model should produce. For Gemini image-generation models (e.g. gemini-2.5-flash-image), pass "IMAGE" to receive image output. Pass "TEXT" and "IMAGE" together to allow mixed output. Leave unset for the provider default (text only).

func WithRetryOnStatus 

func WithRetryOnStatus(status int, retry bool) Option

WithRetryOnStatus overrides whether HTTP-backed providers retry a given status code. By default, providers retry 429 with backoff. Pass WithRetryOnStatus(429, false) to surface rate limits immediately so the caller can apply its own backoff or failover. Pass true to opt into retry for a status that is not retried by default (for example 500).

func WithTemperature 

func WithTemperature(temp float64) Option

WithTemperature sets the sampling temperature.

func WithTokenSource 

func WithTokenSource(source auth.TokenSource) Option

WithTokenSource configures an abstract token source for providers that support OAuth or dynamic token acquisition.

For providers supporting both API keys and OAuth (e.g. xAI), a non-nil TokenSource takes precedence over APIKey.

func WithToolChoice 

func WithToolChoice(choice ToolChoice) Option

WithToolChoice controls whether supplied tools may, must, or must not be used.

func WithTools 

func WithTools(tools ...Tool) Option

WithTools supplies callable tools the model may request.

func WithWebSearch 

func WithWebSearch(enabled bool) Option

WithWebSearch enables web search/grounding capabilities. For Gemini, this enables google_search tool. For Perplexity models, web search is always enabled.

type Provider 

type Provider interface {
	Name() string
	Generate(ctx context.Context, prompt []*Message, opts ...Option) (*Response, error)
	Stream(ctx context.Context, prompt []*Message, opts ...Option) (<-chan StreamChunk, error)
}

Provider describes a backend capable of generating responses from LLMs.

type ProviderFactory 

type ProviderFactory func(apiKey string, opts ...Option) (Provider, error)

ProviderFactory describes a function that can instantiate a provider backed by a specific vendor SDK.

type ReasoningContent 

type ReasoningContent struct {
	Text string
}

ReasoningContent represents model-internal reasoning or thinking text when providers expose it.

func Reasoning 

func Reasoning(s string) *ReasoningContent

Reasoning is a helper constructor for a reasoning part.

func (*ReasoningContent) Type 

func (r *ReasoningContent) Type() string

Type identifies the piece as reasoning.

type Response 

type Response struct {
	ID      string
	Content []ContentPart
	Usage   UsageMetadata
	Raw     interface{}
}

Response contains the normalized result returned from a provider.

func (*Response) ReasoningText 

func (r *Response) ReasoningText() string

ReasoningText concatenates reasoning segments exposed by providers.

func (*Response) Text 

func (r *Response) Text() string

Text concatenates the textual segments of the response for the common use case where only text matters.

func (*Response) ToolCalls 

func (r *Response) ToolCalls() []*ToolCallContent

ToolCalls returns all normalized tool calls requested in the response.

type Role 

type Role string

Role represents the speaker for a given message. 

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type StreamChunk 

type StreamChunk struct {
	// ID is the upstream completion identifier when the provider reports one
	// (OpenAI-compatible lanes and Anthropic). It is best-effort telemetry and
	// may be empty for lanes that do not report a stable stream id.
	ID string
	// Delta is the incremental text produced since the previous chunk.
	Delta string
	// ReasoningDelta is the incremental reasoning text produced since the previous chunk.
	ReasoningDelta string
	// ToolCalls carries streamed tool call deltas.
	ToolCalls []*ToolCallContent
	// Usage carries token and cost accounting when the upstream reports it.
	Usage *UsageMetadata
	// FinishReason describes how the response ended, using the OpenAI
	// vocabulary (stop, length, tool_calls). It is empty when the upstream
	// reports no reason or the reason cannot be mapped. It is typically set on
	// the final content frame (not on the terminal Done frame).
	FinishReason string
	Done         bool
	Err          error
}

StreamChunk represents a partial streaming response.

type TextContent 

type TextContent struct {
	Text string
}

TextContent represents free-form text.

func Text 

func Text(s string) *TextContent

Text is a helper constructor for a text part.

func (*TextContent) Type 

func (t *TextContent) Type() string

Type identifies the piece as text.

type Tool 

type Tool struct {
	Name        string
	Description string
	Parameters  map[string]interface{}
}

Tool describes a callable function the model may request.

func NewTool 

func NewTool(name, description string, parameters map[string]interface{}) Tool

NewTool constructs a callable tool definition.

type ToolCallContent 

type ToolCallContent struct {
	Index     int
	ID        string
	Name      string
	Arguments string
}

ToolCallContent represents a model-requested call to a tool.

func ToolCall 

func ToolCall(id, name, arguments string) *ToolCallContent

ToolCall is a helper constructor for a tool-call content part.

func ToolCallWithIndex 

func ToolCallWithIndex(index int, id, name, arguments string) *ToolCallContent

ToolCallWithIndex is a helper constructor for a streaming tool-call delta.

func (*ToolCallContent) Type 

func (t *ToolCallContent) Type() string

Type identifies the piece as a tool call.

type ToolChoice 

type ToolChoice struct {
	Mode ToolChoiceMode
	Name string
}

ToolChoice controls whether the model may, must, or must not call tools.

func AutoToolChoice 

func AutoToolChoice() ToolChoice

AutoToolChoice lets the model choose whether to call tools.

func NamedToolChoice 

func NamedToolChoice(name string) ToolChoice

NamedToolChoice requires the model to call the named tool.

func NoToolChoice 

func NoToolChoice() ToolChoice

NoToolChoice prevents the model from calling tools.

func RequiredToolChoice 

func RequiredToolChoice() ToolChoice

RequiredToolChoice requires the model to call at least one tool.

type ToolChoiceMode 

type ToolChoiceMode string

ToolChoiceMode controls how providers should use supplied tools. 

const (
	ToolChoiceAuto     ToolChoiceMode = "auto"
	ToolChoiceNone     ToolChoiceMode = "none"
	ToolChoiceRequired ToolChoiceMode = "required"
	ToolChoiceNamed    ToolChoiceMode = "named"
)

type UsageMetadata 

type UsageMetadata struct {
	PromptTokens        int
	CompletionTokens    int
	TotalTokens         int
	CacheReadTokens     int
	CacheCreationTokens int
	ReasoningTokens     int
	Cost                float64 // Estimated cost in US dollars based on configured per-million-token rates.
}

UsageMetadata captures token consumption and cost information reported by providers.

Source Files

View all Source files

Directories

Path Synopsis
oauth2
examples
cli command
CLI tool for end-to-end testing of llmhub providers.
 CLI tool for end-to-end testing of llmhub providers.
xai-oauth command
zai command
zai demonstrates using llmhub's Z.AI provider with the OpenAI-compatible Chat Completions API.
 zai demonstrates using llmhub's Z.AI provider with the OpenAI-compatible Chat Completions API.
internal
httpretry
openaichat
sse
providers
anthropic
gemini
ollama
openai
openrouter
xai
zai

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.