llm

package
v0.0.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package llm provides plain-HTTP clients for OpenAI-compatible Chat Completions and Embeddings APIs.

Client and EmbeddingClient own the wire-level request and response types and can be configured for any compatible endpoint with Options: WithBaseURL, WithAPIKey, WithHeaders, WithQueryParams, WithExtraFields, WithRetry, WithRequestTimeout, WithHTTPClient, WithMaxResponseBytes, and the reasoning options described below.

Configuring providers

There is one client. Configure a provider by composing options: 

// OpenRouter. DefaultBaseURL already points at it.
client := llm.NewClient("openai/gpt-4.1",
	llm.WithAPIKey(os.Getenv("OPENROUTER_API_KEY")),
	llm.WithHeaders(map[string]string{
		"HTTP-Referer": "https://example.com",
		"X-Title":      "My App",
	}),
	llm.WithReasoningDialect(llm.ReasoningOpenRouter),
)

// SGLang or any other self-hosted server. Local deployments commonly
// ignore auth, so a placeholder key works.
client := llm.NewClient(model,
	llm.WithBaseURL("http://localhost:30000/v1"),
	llm.WithAPIKey("local"),
	llm.WithReasoningDialect(llm.ReasoningSGLang),
	llm.WithReasoningEffort("high"),
)

// vLLM.
client := llm.NewClient(model,
	llm.WithBaseURL("http://localhost:8000/v1"),
	llm.WithAPIKey("local"),
	llm.WithReasoningDialect(llm.ReasoningVLLM),
)

WithExtraFields adds static JSON body fields to every request; dotted keys create nested objects. For example, OpenRouter zero data retention is WithExtraFields(map[string]any{"provider.zdr": true}) and provider routing is WithExtraFields(map[string]any{"provider.order": []string{"..."}}).

Embeddings use the same options through NewEmbeddingClient: 

embedder := llm.NewEmbeddingClient(llm.EmbeddingConfig{Model: "baai/bge-m3"},
	llm.WithBaseURL("http://localhost:30000/v1"),
	llm.WithAPIKey("local"),
)

Reasoning

Reasoning-capable providers use incompatible request fields, so reasoning requires a dialect: WithReasoningDialect(ReasoningOpenRouter, ReasoningVLLM, or ReasoningSGLang). Without a dialect, reasoning controls are ignored and requests carry only what extra fields add.

With a dialect set, two controls apply: ChatRequest.ReasoningTokenBudget (a token count) and WithReasoningEffort (a qualitative level). Effort takes precedence over the budget on every dialect. Responses decode reasoning_content (and the legacy "reasoning" field) into ChatMessage.ReasoningContent regardless of dialect.

Wire mapping by dialect: 

Dialect	Budget wire fields	Effort wire fields	Effort values
OpenRouter	reasoning.max_tokens	reasoning.effort	model-dependent, e.g. low, medium, high
vLLM	thinking_token_budget and chat_template_kwargs.enable_thinking	reasoning_effort and chat_template_kwargs.enable_thinking	model- and template-dependent
SGLang	none (ignored)	reasoning_effort	none, minimal, low, medium, high, xhigh, max, or a float in [0, 0.99]

A non-positive budget with no effort configured emits reasoning-off controls on OpenRouter (reasoning.effort "none") and vLLM (reasoning_effort "none" with enable_thinking false). SGLang emits nothing without an effort, so set WithReasoningEffort("none") to disable thinking explicitly. Setting effort to "none" disables reasoning on every dialect. Effort values pass through unvalidated; providers reject values they do not support.

Two caveats worth knowing. vLLM accepts reasoning_effort on the wire but most chat templates ignore it, so the dialect pairs effort with the chat_template_kwargs.enable_thinking toggle, which is the dependable control. SGLang instead propagates reasoning_effort into per-model chat-template kwargs server-side, so effort alone controls reasoning across model families.

OpenAI's API also accepts a top-level reasoning_effort but has no dialect here. Use a dialect-less client with WithExtraFields(map[string]any{"reasoning_effort": "high"}), or compose extra fields for any other provider the dialects do not cover.

Retries and failover

Retries are disabled by default. RetryConfig and WithRetry enable bounded, context-aware retries for transient transport, response-read, and provider status errors.

CircuitBreaker and EmbeddingCircuitBreaker add ordered provider failover. They classify transport and provider errors, temporarily bypass an unavailable primary, and periodically probe it for recovery.

This package does not depend on the agent runtime. Client and CircuitBreaker satisfy agent.Model without requiring an adapter.

Index

Examples

Constants

View Source 
const (
	// DefaultBaseURL is the OpenRouter OpenAI-compatible API base URL.
	DefaultBaseURL = "https://openrouter.ai/api/v1"
	// DefaultRequestTimeout is the default maximum duration of one provider request.
	DefaultRequestTimeout = 60 * time.Second
	// DefaultMaxResponseBytes caps response bodies at 64 MiB. This is generous
	// for current OpenAI-compatible max completion sizes while still preventing
	// accidental unbounded reads. Use WithMaxResponseBytes(0) to disable it.
	DefaultMaxResponseBytes = 64 << 20
)
View Source 
const (
	// DefaultRetryDelay is the base delay used when retries are enabled.
	DefaultRetryDelay = 100 * time.Millisecond
	// DefaultRetryMaxJitter is the maximum random delay added by the default strategy.
	DefaultRetryMaxJitter = 100 * time.Millisecond
)

Variables

This section is empty.

Functions

func BackOffDelay added in v0.0.7 

func BackOffDelay(attempt uint, _ error, config DelayContext) time.Duration

BackOffDelay returns an exponentially increasing delay.

func FixedDelay added in v0.0.7 

func FixedDelay(_ uint, _ error, config DelayContext) time.Duration

FixedDelay returns the configured base delay.

func FullJitterBackoffDelay added in v0.0.7 

func FullJitterBackoffDelay(attempt uint, _ error, config DelayContext) time.Duration

FullJitterBackoffDelay returns a random delay below an exponential ceiling.

func RandomDelay added in v0.0.7 

func RandomDelay(_ uint, _ error, config DelayContext) time.Duration

RandomDelay returns a random delay below the configured maximum jitter.

Types

type BreakerConfig 

type BreakerConfig struct {
	// FailureThreshold is the number of consecutive counted primary failures
	// required to open the circuit. Values below one use the package default.
	FailureThreshold int
	// OpenDuration is how long an opened circuit bypasses the primary before a
	// recovery probe may run.
	OpenDuration time.Duration
	// ProbeInterval controls the delay between recovery probes.
	ProbeInterval time.Duration
	// SuccessfulProbeThreshold is the number of consecutive successful probes
	// required to close the circuit.
	SuccessfulProbeThreshold int
	// OnFailover receives each transition from a failed provider to a fallback.
	OnFailover func(context.Context, FailoverEvent)
}

BreakerConfig configures primary-provider circuit breaking. Callbacks run synchronously outside the breaker lock and must be safe for concurrent use.

type ChatChoice 

type ChatChoice struct {
	Message      ChatMessage `json:"message"`
	FinishReason string      `json:"finish_reason"`
}

ChatChoice is a single Chat Completions choice.

type ChatFunctionCall 

type ChatFunctionCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ChatFunctionCall is a tool/function call payload in a chat message.

type ChatFunctionDefinition 

type ChatFunctionDefinition struct {
	Name        string          `json:"name"`
	Description string          `json:"description,omitempty"`
	Parameters  json.RawMessage `json:"parameters"`
	Strict      bool            `json:"strict,omitempty"`
}

ChatFunctionDefinition describes a function tool.

type ChatMessage 

type ChatMessage struct {
	Role             string         `json:"role"`
	Content          string         `json:"content,omitempty"`
	ReasoningContent string         `json:"reasoning_content,omitempty"`
	Refusal          string         `json:"refusal,omitempty"`
	ToolCalls        []ChatToolCall `json:"tool_calls,omitempty"`
	ToolCallID       string         `json:"tool_call_id,omitempty"`
}

ChatMessage is an OpenAI-compatible chat message. ReasoningContent and Refusal are response fields exposed by compatible providers.

func (ChatMessage) MarshalJSON 

func (m ChatMessage) MarshalJSON() ([]byte, error)

func (*ChatMessage) UnmarshalJSON 

func (m *ChatMessage) UnmarshalJSON(data []byte) error

type ChatRequest 

type ChatRequest struct {
	Messages    []ChatMessage `json:"messages"`
	Tools       []ChatTool    `json:"tools,omitempty"`
	Temperature *float64      `json:"temperature,omitempty"`
	TopP        *float64      `json:"top_p,omitempty"`
	MaxTokens   *int          `json:"max_completion_tokens,omitempty"`
	// ReasoningTokenBudget enables reasoning when greater than zero and a
	// reasoning dialect is configured. ReasoningOpenRouter and ReasoningVLLM
	// translate non-positive values into provider-specific reasoning-off
	// payloads; ReasoningSGLang ignores the budget in favor of
	// WithReasoningEffort.
	ReasoningTokenBudget int               `json:"-"`
	ParallelToolCalls    *bool             `json:"parallel_tool_calls,omitempty"`
	ToolChoice           any               `json:"tool_choice,omitempty"`
	StructuredOutput     *StructuredOutput `json:"-"`
	// RequestTimeout overrides the client timeout for each provider attempt.
	// Zero uses the client default; a negative duration disables it.
	RequestTimeout time.Duration `json:"-"`
}

ChatRequest is an OpenAI-compatible Chat Completions request without the model field. Client injects its configured model into each request.

type ChatResponse 

type ChatResponse struct {
	ID      string       `json:"id"`
	Choices []ChatChoice `json:"choices"`
	Usage   ChatUsage    `json:"usage"`
}

ChatResponse is an OpenAI-compatible Chat Completions response.

type ChatTool 

type ChatTool struct {
	Type     string                 `json:"type"`
	Function ChatFunctionDefinition `json:"function"`
}

ChatTool is an OpenAI-compatible tool definition.

type ChatToolCall 

type ChatToolCall struct {
	ID       string           `json:"id"`
	Type     string           `json:"type"`
	Function ChatFunctionCall `json:"function"`
}

ChatToolCall is an OpenAI-compatible tool call.

type ChatUsage 

type ChatUsage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

ChatUsage is token usage returned by the Chat Completions API.

type CircuitBreaker 

type CircuitBreaker struct {
	// contains filtered or unexported fields
}

CircuitBreaker routes Chat Completions requests through a primary client and optional fallbacks. With no fallbacks it is a pass-through wrapper. The breaker records errors that indicate the primary provider is unavailable.

func NewCircuitBreaker 

func NewCircuitBreaker(primary *Client, fallbacks ...*Client) *CircuitBreaker

NewCircuitBreaker builds a CircuitBreaker around primary and optional fallbacks. The primary client is assumed to be non-nil.

func NewCircuitBreakerWithConfig 

func NewCircuitBreakerWithConfig(cfg BreakerConfig, primary *Client, fallbacks ...*Client) *CircuitBreaker

NewCircuitBreakerWithConfig builds a CircuitBreaker with custom breaker settings. It is primarily useful for tests and tuned production failover.

func (*CircuitBreaker) Complete 

func (b *CircuitBreaker) Complete(ctx context.Context, req ChatRequest) (*ChatResponse, error)

Complete sends a Chat Completions request. Provider failures fall through to configured fallbacks after the client's configured retries are exhausted.

Example 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	primary := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		http.Error(w, "temporarily unavailable", http.StatusServiceUnavailable)
	}))
	defer primary.Close()
	fallback := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message:      llm.ChatMessage{Role: "assistant", Content: "served by fallback"},
				FinishReason: "stop",
			}},
		})
	}))
	defer fallback.Close()

	breaker := llm.NewCircuitBreaker(
		llm.NewClient("model", llm.WithProviderID("primary"), llm.WithBaseURL(primary.URL)),
		llm.NewClient("model", llm.WithProviderID("fallback"), llm.WithBaseURL(fallback.URL)),
	)
	response, err := breaker.Complete(context.Background(), llm.ChatRequest{
		Messages: []llm.ChatMessage{{Role: "user", Content: "Hello"}},
	})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(response.Choices[0].Message.Content)

}

Output:
served by fallback

func (*CircuitBreaker) ContextWindow added in v0.0.9 

func (b *CircuitBreaker) ContextWindow(ctx context.Context) (int, error)

ContextWindow returns the smallest positive context window known across the primary and fallback providers. Providers whose window cannot be discovered are ignored when another provider is known. If no provider reports a window, the joined discovery errors are returned so callers can remain reactive.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client calls an OpenAI-compatible Chat Completions API.

func NewClient 

func NewClient(model string, opts ...Option) *Client

NewClient builds a Chat Completions client for model. By default it uses DefaultBaseURL.

Example (OpenRouter) 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var request map[string]any
		if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
			panic(err)
		}
		provider := request["provider"].(map[string]any)
		fmt.Println(r.Header.Get("HTTP-Referer"))
		fmt.Println(r.Header.Get("X-Title"))
		fmt.Println(provider["zdr"])
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message:      llm.ChatMessage{Role: "assistant", Content: "ok"},
				FinishReason: "stop",
			}},
		})
	}))
	defer server.Close()

	client := llm.NewClient("openrouter/model",
		llm.WithAPIKey("example-key"),
		llm.WithBaseURL(server.URL),
		llm.WithHeaders(map[string]string{
			"HTTP-Referer": "https://example.com",
			"X-Title":      "Example App",
		}),
		llm.WithExtraFields(map[string]any{"provider.zdr": true}),
		llm.WithReasoningDialect(llm.ReasoningOpenRouter),
	)
	_, err := client.Complete(context.Background(), llm.ChatRequest{
		Messages: []llm.ChatMessage{{Role: "user", Content: "Hello"}},
	})
	if err != nil {
		fmt.Println(err)
	}

}

Output:
https://example.com
Example App
true

func (*Client) Complete 

func (c *Client) Complete(ctx context.Context, req ChatRequest) (*ChatResponse, error)

Complete sends a Chat Completions request.

Example 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var request struct {
			Model    string            `json:"model"`
			Messages []llm.ChatMessage `json:"messages"`
		}
		if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
			panic(err)
		}
		fmt.Printf("%s: %s\n", request.Model, request.Messages[0].Content)
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message:      llm.ChatMessage{Role: "assistant", Content: "Hello!"},
				FinishReason: "stop",
			}},
		})
	}))
	defer server.Close()

	client := llm.NewClient("example-model",
		llm.WithBaseURL(server.URL),
		llm.WithAPIKey("example-key"),
	)
	response, err := client.Complete(context.Background(), llm.ChatRequest{
		Messages: []llm.ChatMessage{{Role: "user", Content: "Say hello."}},
	})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(response.Choices[0].Message.Content)

}

Output:
example-model: Say hello.
Hello!
Example (WithReasoning) 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var request struct {
			Reasoning struct {
				MaxTokens int `json:"max_tokens"`
			} `json:"reasoning"`
		}
		if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
			panic(err)
		}
		fmt.Println("reasoning budget:", request.Reasoning.MaxTokens)
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message: llm.ChatMessage{
					Role:             "assistant",
					Content:          "The answer is 4.",
					ReasoningContent: "2+2 equals 4.",
				},
				FinishReason: "stop",
			}},
		})
	}))
	defer server.Close()

	client := llm.NewClient("example-model",
		llm.WithBaseURL(server.URL),
		llm.WithReasoningDialect(llm.ReasoningOpenRouter),
	)
	response, err := client.Complete(context.Background(), llm.ChatRequest{
		Messages:             []llm.ChatMessage{{Role: "user", Content: "What is 2+2?"}},
		ReasoningTokenBudget: 512,
	})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(response.Choices[0].Message.ReasoningContent)
	fmt.Println(response.Choices[0].Message.Content)

}

Output:
reasoning budget: 512
2+2 equals 4.
The answer is 4.
Example (WithReasoningEffort) 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var request struct {
			ReasoningEffort string `json:"reasoning_effort"`
		}
		if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
			panic(err)
		}
		fmt.Println("reasoning effort:", request.ReasoningEffort)
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message: llm.ChatMessage{
					Role:             "assistant",
					Content:          "The answer is 4.",
					ReasoningContent: "2+2 equals 4.",
				},
				FinishReason: "stop",
			}},
		})
	}))
	defer server.Close()

	client := llm.NewClient("example-model",
		llm.WithBaseURL(server.URL),
		llm.WithReasoningDialect(llm.ReasoningSGLang),
		llm.WithReasoningEffort("high"),
	)
	response, err := client.Complete(context.Background(), llm.ChatRequest{
		Messages: []llm.ChatMessage{{Role: "user", Content: "What is 2+2?"}},
	})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(response.Choices[0].Message.ReasoningContent)
	fmt.Println(response.Choices[0].Message.Content)

}

Output:
reasoning effort: high
2+2 equals 4.
The answer is 4.
Example (WithRetry) 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	attempts := 0
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		attempts++
		if attempts == 1 {
			http.Error(w, "temporarily unavailable", http.StatusServiceUnavailable)
			return
		}
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message:      llm.ChatMessage{Role: "assistant", Content: "Recovered"},
				FinishReason: "stop",
			}},
		})
	}))
	defer server.Close()

	client := llm.NewClient("example-model",
		llm.WithBaseURL(server.URL),
		llm.WithRetry(llm.RetryConfig{
			Attempts:  2,
			Delay:     time.Millisecond,
			DelayType: llm.FixedDelay,
		}),
	)
	response, err := client.Complete(context.Background(), llm.ChatRequest{})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(attempts)
	fmt.Println(response.Choices[0].Message.Content)

}

Output:
2
Recovered

func (*Client) ContextWindow added in v0.0.9 

func (c *Client) ContextWindow(ctx context.Context) (int, error)

ContextWindow returns the configured model's maximum context window in tokens, discovered through the Models API. Successful positive results are cached; transient discovery failures are not, so a later call can succeed. Concurrent discoveries converge on a single published value.

func (*Client) Models 

func (c *Client) Models(ctx context.Context) (*ModelsResponse, error)

Models returns the models advertised by the configured OpenAI-compatible API.

Example 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		writeJSON(w, llm.ModelsResponse{
			Object: "list",
			Data: []llm.ModelInfo{
				{ID: "small-model"},
				{ID: "large-model"},
			},
		})
	}))
	defer server.Close()

	client := llm.NewClient("unused", llm.WithBaseURL(server.URL))
	models, err := client.Models(context.Background())
	if err != nil {
		fmt.Println(err)
		return
	}
	for _, model := range models.Data {
		fmt.Println(model.ID)
	}

}

Output:
small-model
large-model

type ContextOverflowError added in v0.0.9 

type ContextOverflowError struct {
	// Err is the wrapped provider error, typically a *StatusError.
	Err error
}

ContextOverflowError reports that the provider rejected the request because the input exceeded the model's context window. It wraps the underlying provider error so callers can still inspect the original status.

func (*ContextOverflowError) Error added in v0.0.9 

func (e *ContextOverflowError) Error() string

func (*ContextOverflowError) Unwrap added in v0.0.9 

func (e *ContextOverflowError) Unwrap() error

type DelayContext added in v0.0.7 

type DelayContext interface {
	Delay() time.Duration
	MaxDelay() time.Duration
	MaxJitter() time.Duration
}

DelayContext exposes retry settings to a delay strategy.

type DelayTypeFunc added in v0.0.7 

type DelayTypeFunc func(attempt uint, err error, config DelayContext) time.Duration

DelayTypeFunc computes the wait before the next request. Attempt starts at one after the initial request fails.

func CombineDelay added in v0.0.7 

func CombineDelay(delayTypes ...DelayTypeFunc) DelayTypeFunc

CombineDelay adds the results of multiple delay strategies.

type Embedding 

type Embedding struct {
	Object    string    `json:"object"`
	Index     int       `json:"index"`
	Embedding []float32 `json:"embedding"`
}

Embedding is one vector returned by the Embeddings API.

type EmbeddingCircuitBreaker 

type EmbeddingCircuitBreaker struct {
	// contains filtered or unexported fields
}

EmbeddingCircuitBreaker routes embedding requests through a primary client and optional fallbacks.

func NewEmbeddingCircuitBreaker 

func NewEmbeddingCircuitBreaker(primary *EmbeddingClient, fallbacks ...*EmbeddingClient) *EmbeddingCircuitBreaker

NewEmbeddingCircuitBreaker builds an embedding breaker around primary and optional fallbacks.

func NewEmbeddingCircuitBreakerWithConfig 

func NewEmbeddingCircuitBreakerWithConfig(cfg BreakerConfig, primary *EmbeddingClient, fallbacks ...*EmbeddingClient) *EmbeddingCircuitBreaker

NewEmbeddingCircuitBreakerWithConfig builds an embedding breaker with custom breaker settings.

func (*EmbeddingCircuitBreaker) Embed 

func (b *EmbeddingCircuitBreaker) Embed(ctx context.Context, req EmbeddingRequest) (*EmbeddingResponse, error)

Embed creates embeddings using the primary client and optional fallbacks.

type EmbeddingClient 

type EmbeddingClient struct {
	// contains filtered or unexported fields
}

EmbeddingClient calls an OpenAI-compatible Embeddings API.

func NewEmbeddingClient 

func NewEmbeddingClient(cfg EmbeddingConfig, opts ...Option) *EmbeddingClient

NewEmbeddingClient builds an Embeddings client.

func (*EmbeddingClient) Embed 

func (c *EmbeddingClient) Embed(ctx context.Context, req EmbeddingRequest) (*EmbeddingResponse, error)

Embed creates float-encoded embeddings.

Example 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var request struct {
			Model string   `json:"model"`
			Input []string `json:"input"`
		}
		if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
			panic(err)
		}
		fmt.Printf("%s: %d input\n", request.Model, len(request.Input))
		writeJSON(w, llm.EmbeddingResponse{
			Object: "list",
			Model:  request.Model,
			Data: []llm.Embedding{{
				Object:    "embedding",
				Index:     0,
				Embedding: []float32{0.25, 0.75},
			}},
		})
	}))
	defer server.Close()

	client := llm.NewEmbeddingClient(
		llm.EmbeddingConfig{Model: "embedding-model"},
		llm.WithBaseURL(server.URL),
	)
	response, err := client.Embed(context.Background(), llm.EmbeddingRequest{
		Input: []string{"embed this"},
	})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(response.Data[0].Embedding)

}

Output:
embedding-model: 1 input
[0.25 0.75]

type EmbeddingConfig 

type EmbeddingConfig struct {
	Model string
}

EmbeddingConfig identifies an embedding model.

type EmbeddingRequest 

type EmbeddingRequest struct {
	Input      []string `json:"input"`
	Dimensions *int     `json:"dimensions,omitempty"`
	User       string   `json:"user,omitempty"`
	// RequestTimeout overrides the client timeout for each provider attempt.
	// Zero uses the client default; a negative duration disables it.
	RequestTimeout time.Duration `json:"-"`
}

EmbeddingRequest is an OpenAI-compatible text embeddings request without the model field. EmbeddingClient injects its configured model.

type EmbeddingResponse 

type EmbeddingResponse struct {
	Object string         `json:"object"`
	Data   []Embedding    `json:"data"`
	Model  string         `json:"model"`
	Usage  EmbeddingUsage `json:"usage"`
}

EmbeddingResponse is an OpenAI-compatible Embeddings response.

type EmbeddingUsage 

type EmbeddingUsage struct {
	PromptTokens int `json:"prompt_tokens"`
	TotalTokens  int `json:"total_tokens"`
}

EmbeddingUsage is token usage returned by the Embeddings API.

type FailoverEvent 

type FailoverEvent struct {
	// FromProvider is the configured ID of the provider that failed.
	FromProvider string
	// FromProviderIndex is zero for the primary and one-based for fallbacks.
	FromProviderIndex int
	// ToProvider is the configured ID of the next provider to be attempted.
	ToProvider string
	// Error is the provider error that caused failover.
	Error error
	// CountedAgainstBreaker reports whether Error affected primary circuit state.
	CountedAgainstBreaker bool
}

FailoverEvent describes an error-triggered transition to another provider.

type ModelInfo 

type ModelInfo struct {
	ID      string `json:"id"`
	Object  string `json:"object,omitempty"`
	Created int64  `json:"created,omitempty"`
	OwnedBy string `json:"owned_by,omitempty"`
	// ContextLength is the OpenRouter context window in tokens.
	ContextLength int `json:"context_length,omitempty"`
	// MaxModelLen is the vLLM context window in tokens.
	MaxModelLen int `json:"max_model_len,omitempty"`
}

ModelInfo describes one model returned by an OpenAI-compatible Models API.

type ModelsResponse 

type ModelsResponse struct {
	Object string      `json:"object"`
	Data   []ModelInfo `json:"data"`
}

ModelsResponse is the OpenAI-compatible Models API response.

type Option 

type Option func(*Client)

Option customizes a Client.

func WithAPIKey 

func WithAPIKey(key string) Option

WithAPIKey overrides the bearer token used for API requests.

func WithBaseURL 

func WithBaseURL(baseURL string) Option

WithBaseURL overrides the API base URL, for example to point at vLLM or another OpenAI-compatible server.

func WithExtraFields 

func WithExtraFields(extraFields map[string]any) Option

WithExtraFields adds static JSON body fields to every request made by the client. Dotted keys create nested objects, for example "provider.zdr".

func WithHTTPClient 

func WithHTTPClient(httpClient *http.Client) Option

WithHTTPClient overrides the HTTP client used to execute requests.

func WithHeaders 

func WithHeaders(headers map[string]string) Option

WithHeaders adds static headers to every request made by the client.

func WithMaxResponseBytes 

func WithMaxResponseBytes(n int64) Option

WithMaxResponseBytes overrides the maximum response body size read by the client. Zero or a negative value disables the limit.

func WithProviderID 

func WithProviderID(providerID string) Option

WithProviderID identifies the provider in circuit-breaker failover events.

func WithQueryParams 

func WithQueryParams(queryParams map[string]string) Option

WithQueryParams adds static query parameters to every request made by the client.

func WithReasoningDialect added in v0.0.8 

func WithReasoningDialect(dialect ReasoningDialect) Option

WithReasoningDialect selects the provider-specific wire format for reasoning controls. Without a dialect, reasoning options are ignored.

func WithReasoningEffort added in v0.0.8 

func WithReasoningEffort(effort string) Option

WithReasoningEffort sets a qualitative reasoning level, for example "none", "low", "medium", or "high". Providers define the accepted values; the effort is passed through unvalidated. Effort takes precedence over ReasoningTokenBudget and requires a reasoning dialect.

func WithRequestTimeout 

func WithRequestTimeout(timeout time.Duration) Option

WithRequestTimeout overrides the client's default maximum duration of one provider request. Zero or a negative duration disables the client default.

func WithRetry added in v0.0.7 

func WithRetry(config RetryConfig) Option

WithRetry configures optional retries for a client.

type ReasoningDialect added in v0.0.8 

type ReasoningDialect uint8

ReasoningDialect selects the provider-specific wire format used to enable, disable, and size reasoning. The zero value emits no reasoning controls; reasoning options are ignored without a dialect. 

const (
	// ReasoningOpenRouter targets OpenRouter: a "reasoning" object with either
	// "max_tokens" (token budget) or "effort" (qualitative level).
	ReasoningOpenRouter ReasoningDialect = iota + 1
	// ReasoningVLLM targets vLLM: "thinking_token_budget" plus
	// "chat_template_kwargs.enable_thinking", or "reasoning_effort".
	ReasoningVLLM
	// ReasoningSGLang targets SGLang: top-level "reasoning_effort". SGLang has
	// no token budget parameter, so ReasoningTokenBudget is ignored and only
	// WithReasoningEffort controls reasoning.
	ReasoningSGLang
)

Supported reasoning dialects.

type ResponseTooLargeError 

type ResponseTooLargeError struct {
	Limit int64
	// contains filtered or unexported fields
}

ResponseTooLargeError reports a response body that exceeded the configured maximum response size.

func (*ResponseTooLargeError) Error 

func (e *ResponseTooLargeError) Error() string

type RetryConfig added in v0.0.7 

type RetryConfig struct {
	Attempts  uint
	Delay     time.Duration
	MaxDelay  time.Duration
	MaxJitter time.Duration
	DelayType DelayTypeFunc
}

RetryConfig configures optional retries for one client. Attempts counts the initial request; values below two disable retries. A zero Delay or MaxJitter uses its package default, while a negative value disables it. A non-positive MaxDelay leaves delays uncapped.

type StatusError 

type StatusError struct {
	StatusCode int
	Body       string
	// contains filtered or unexported fields
}

StatusError reports a non-200 response from a provider API.

func (*StatusError) Error 

func (e *StatusError) Error() string

type StructuredOutput 

type StructuredOutput struct {
	Name        string          `json:"name"`
	Description string          `json:"description,omitempty"`
	Schema      json.RawMessage `json:"schema"`
	Strict      bool            `json:"strict,omitempty"`
}

StructuredOutput constrains a chat completion to a JSON Schema. Providers must support OpenAI-compatible strict structured outputs.

Example: 

request := llm.ChatRequest{
	Messages: []llm.ChatMessage{{Role: "user", Content: "Give me an answer."}},
	StructuredOutput: &llm.StructuredOutput{
		Name:   "answer",
		Schema: json.RawMessage(`{"type":"object","properties":{"answer":{"type":"string"}},"required":["answer"],"additionalProperties":false}`),
		Strict: true,
	},
}
Example 
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/DavidNix/safeagent/go/llm"
)

func writeJSON(w http.ResponseWriter, value any) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(value); err != nil {
		panic(err)
	}
}

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var request struct {
			ResponseFormat struct {
				Type       string               `json:"type"`
				JSONSchema llm.StructuredOutput `json:"json_schema"`
			} `json:"response_format"`
		}
		if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
			panic(err)
		}
		fmt.Println(request.ResponseFormat.Type)
		fmt.Println(request.ResponseFormat.JSONSchema.Name)
		writeJSON(w, llm.ChatResponse{
			Choices: []llm.ChatChoice{{
				Message:      llm.ChatMessage{Role: "assistant", Content: `{"answer":"42"}`},
				FinishReason: "stop",
			}},
		})
	}))
	defer server.Close()

	client := llm.NewClient("example-model", llm.WithBaseURL(server.URL))
	response, err := client.Complete(context.Background(), llm.ChatRequest{
		Messages: []llm.ChatMessage{{Role: "user", Content: "What is the answer?"}},
		StructuredOutput: &llm.StructuredOutput{
			Name:   "answer",
			Schema: json.RawMessage(`{"type":"object","properties":{"answer":{"type":"string"}},"required":["answer"],"additionalProperties":false}`),
			Strict: true,
		},
	})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(response.Choices[0].Message.Content)

}

Output:
json_schema
answer
{"answer":"42"}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.