client

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 35 Imported by: 0

Documentation

Overview

Package client provides LLM provider clients for Anthropic, OpenAI, and OpenAI-compatible APIs with streaming, retry, and provider detection.

Package client provides request coalescing for identical concurrent LLM requests. When multiple goroutines send identical requests simultaneously (same provider, model, messages, temperature, max_tokens), the Coalescer deduplicates them into a single API call and broadcasts the result to all waiters.

Index

Examples

Constants

View Source 
const (
	GuardrailPII             = core.GuardrailPII
	GuardrailPromptInjection = core.GuardrailPromptInjection
	GuardrailHarmfulContent  = core.GuardrailHarmfulContent
	GuardrailSecretLeak      = core.GuardrailSecretLeak
	GuardrailCustom          = core.GuardrailCustom
	GuardrailBlock           = core.GuardrailBlock
	GuardrailRedact          = core.GuardrailRedact
	GuardrailWarn            = core.GuardrailWarn
	SeverityLow              = core.SeverityLow
	SeverityMedium           = core.SeverityMedium
	SeverityHigh             = core.SeverityHigh
	SeverityCritical         = core.SeverityCritical
)

Guardrail rule types, actions, and severities (see core).

View Source 
const (
	ResponseOK                 = core.ResponseOK
	ResponseErrorOnlyReasoning = core.ResponseErrorOnlyReasoning
	ResponseEmpty              = core.ResponseEmpty
	ResponseMalformedStream    = core.ResponseMalformedStream
)

Response health classifications (see core.DetectResponseHealth).

View Source 
const (
	ChatProtocolCompletions = adapters.ChatProtocolCompletions
	ChatProtocolMessages    = adapters.ChatProtocolMessages
)

Adapter protocol constants.

View Source 
const (
	ProviderTypeAnthropic        = adapters.ProviderTypeAnthropic
	ProviderTypeOpenAI           = adapters.ProviderTypeOpenAI
	ProviderTypeOpenAICompatible = adapters.ProviderTypeOpenAICompatible
	ProviderTypeAzure            = adapters.ProviderTypeAzure
	ProviderTypeBedrock          = adapters.ProviderTypeBedrock
	ProviderTypeVertex           = adapters.ProviderTypeVertex
)

Provider registry constants.

View Source 
const (
	// RolePrimary is the default, highest-capability model slot.
	RolePrimary = "primary"
	// RoleWeak is a cheaper/faster model used for auxiliary work such as
	// summarization or classification.
	RoleWeak = "weak"
	// RoleEditor is a model used to revise or refine prior output.
	RoleEditor = "editor"
)

Model role slot names. These identify a logical role that a concrete model fills, letting callers route a request to the appropriate model without hard-coding model ids at the call site.

View Source 
const (
	ThinkingFormatNone       = ""
	ThinkingFormatZAI        = "zai"
	ThinkingFormatLongCat    = "longcat"
	ThinkingFormatKimi       = "kimi"
	ThinkingFormatDeepSeek   = "deepseek"
	ThinkingFormatXiaomi     = "xiaomi"
	ThinkingFormatMiniMax    = "minimax" // MiniMax OpenAI: thinking={"type":"adaptive"|"disabled"}
	ThinkingFormatAgnes      = "agnes"
	ThinkingFormatQwen       = "qwen"
	ThinkingFormatOpenRouter = "openrouter"
)

ProviderThinkingFormat is the wire encoding for extended thinking / reasoning. Each provider that supports a host-controlled toggle has its own format; the host preference is always the generic ChatOptions.ThinkingEnabled (with deprecated GLMThinkingEnabled as a Z.AI-era alias).

Formats (from official provider docs): 

zai / longcat / kimi / deepseek / xiaomi → thinking={"type":"enabled"|"disabled"}
minimax                                    → thinking={"type":"adaptive"|"disabled"}
agnes                                      → chat_template_kwargs.enable_thinking
qwen                                       → enable_thinking (top-level)
openrouter                                 → reasoning={enabled: true|false}

Variables

View Source 
var (
	CoreProviders             = adapters.CoreProviders
	OpenAICompatibleProviders = adapters.OpenAICompatibleProviders
)
View Source 
var (
	OpenAICompat      = adapters.OpenAICompat
	GrokCompat        = adapters.GrokCompat
	OpenRouterCompat  = adapters.OpenRouterCompat
	GeminiCompat      = adapters.GeminiCompat
	ZAICompat         = adapters.ZAICompat
	CanopyWaveCompat  = adapters.CanopyWaveCompat
	OpenGatewayCompat = adapters.OpenGatewayCompat
	OllamaCompat      = adapters.OllamaCompat
	OpenCodeGoCompat  = adapters.OpenCodeGoCompat
	PoolsideCompat    = adapters.PoolsideCompat
	GroqCompat        = adapters.GroqCompat
	ClinePassCompat   = adapters.ClinePassCompat
	KimiCompat        = adapters.KimiCompat
	XiaomiCompat      = adapters.XiaomiCompat
	AzureCompat       = adapters.AzureCompat
	BedrockCompat     = adapters.BedrockCompat
	VertexCompat      = adapters.VertexCompat
	DeepSeekCompat    = adapters.DeepSeekCompat
	AgnesCompat       = adapters.AgnesCompat
	LongCatCompat     = adapters.LongCatCompat
	StepFunCompat     = adapters.StepFunCompat
	MiniMaxCompat     = adapters.MiniMaxCompat
)

Per-provider compat configs.

View Source 
var ErrBudgetExceeded = errors.New("graycode-router: virtual key budget exceeded")

ErrBudgetExceeded is returned when a virtual key has exhausted its budget.

View Source 
var ErrResultNotVisible = errors.New("graycode-router: batch result row not yet visible")

ErrResultNotVisible reports that a request's result row was absent even though polling continued past batch completion — callers may retry.

View Source 
var ErrUnknownVirtualKey = errors.New("graycode-router: unknown virtual key")

ErrUnknownVirtualKey is returned when a request references a virtual key that the store does not recognize.

View Source 
var Version = "dev"

Version mirrors core.Version for backward compatibility; the canonical value lives in client/core so subpackages can build User-Agent strings. Default is "dev" until the root package initialises.

Functions

func ActualCostUSD 

func ActualCostUSD(model string, usage *GraycodeRouterUsage) float64

ActualCostUSD computes the realized USD cost of a completed call from token usage using the same per-model pricing as the cost estimator.

func AnthropicBaseFromOpenAIV1 

func AnthropicBaseFromOpenAIV1(openAIBase string) string

func ApplyRedactions 

func ApplyRedactions(response string, violations []GuardrailViolation) string

ApplyRedactions scrubs matched content from a response.

func CloseIdleConnections 

func CloseIdleConnections()

CloseIdleConnections closes idle pooled connections across all provider clients.

func DetectProvider 

func DetectProvider() string

DetectProvider detects the active provider from the credential store.

func EffectiveThinkingEnabled 

func EffectiveThinkingEnabled(opts core.ChatOptions) *bool

EffectiveThinkingEnabled returns the host thinking preference. ThinkingEnabled is the standard field; GLMThinkingEnabled is accepted as a deprecated alias so older Z.AI call sites keep working.

func FormatUsageBar 

func FormatUsageBar(pct float64, width int) string

func FreezeRegistry 

func FreezeRegistry()

FreezeRegistry prevents further provider registrations.

func IsContextOverflow 

func IsContextOverflow(err error) bool

IsContextOverflow reports whether an error means the request exceeded the selected model's context window.

func NewPooledHTTPClient 

func NewPooledHTTPClient(timeout time.Duration) *http.Client

NewPooledHTTPClient returns an *http.Client sharing the pooled transport.

func NormalizeThinkingOptions 

func NormalizeThinkingOptions(opts core.ChatOptions) core.ChatOptions

NormalizeThinkingOptions copies ThinkingEnabled ↔ GLMThinkingEnabled so both fields stay consistent for adapters and older hosts.

func ParseCustomHeaders 

func ParseCustomHeaders() map[string]string

ParseCustomHeaders parses GRAYCODE_CUSTOM_HEADERS env var into a map.

func ProviderSupportsThinkingToggle 

func ProviderSupportsThinkingToggle(provider string) bool

ProviderSupportsThinkingToggle reports whether the provider honors the generic ThinkingEnabled host preference on the wire.

func RegisterDynamicProvider 

func RegisterDynamicProvider(name, baseURL, envKey string) error

RegisterDynamicProvider adds a user-defined OpenAI-compatible provider at runtime.

func ResolveDefaultModel 

func ResolveDefaultModel(provider string) string

ResolveDefaultModel resolves the default model for a provider from the catalog.

func ResolveProviderModelEnvOverride 

func ResolveProviderModelEnvOverride(provider string) string

ResolveProviderModelEnvOverride resolves the model env override for a provider.

func ResolveRole 

func ResolveRole(roles ModelRoles, role string) string

ResolveRole returns the model id configured for the given role, defaulting to Primary when the requested role is unknown or its slot is empty. An empty Primary returns "" so the caller's existing default-model logic still applies.

func ResponseHasContent 

func ResponseHasContent(resp *GraycodeRouterResponse) bool

ResponseHasContent reports whether a response carries content or tool calls.

func RoleFromContext 

func RoleFromContext(ctx context.Context) string

RoleFromContext extracts the role from the context, if present.

func SaveCassette 

func SaveCassette(c *Cassette, path string) error

SaveCassette writes a cassette to a JSON file atomically (temp file + rename).

func SetVersion 

func SetVersion(v string)

SetVersion is called by the root graycode-router package's init to wire the canonical version from the VERSION file into this sub-package (and client/core).

func ValidateStructuredOutput 

func ValidateStructuredOutput(response string, schema map[string]interface{}) error

ValidateStructuredOutput validates a JSON response against a schema. It checks that the response is valid JSON and that all required fields specified in the schema are present with correct types.

func VirtualKeyFromContext 

func VirtualKeyFromContext(ctx context.Context) string

VirtualKeyFromContext extracts a virtual key id from the context, if present.

func WithRole 

func WithRole(ctx context.Context, role string) context.Context

WithRole returns a context carrying the named role for a call. The RoleRouter reads this to select the model for the request.

func WithVirtualKey 

func WithVirtualKey(ctx context.Context, id string) context.Context

WithVirtualKey returns a context carrying the given virtual key id. The BudgetProvider reads this (falling back to ChatOptions.VirtualKeyID) to attribute and enforce spend.

Types

type AdaptiveRateLimitConfig 

type AdaptiveRateLimitConfig struct {
	// RPMLimit is the maximum requests per minute (0 = no limit).
	RPMLimit int
	// TPMLimit is the maximum tokens per minute (0 = no limit).
	TPMLimit int
	// ThresholdPercent is the percentage of remaining quota below which
	// the provider starts throttling. Default is 10 (i.e., <10% remaining).
	ThresholdPercent int
	// MaxDelay is the maximum time to delay when throttling.
	// Default is 10 seconds. Set to 0 to return an error instead of delaying.
	MaxDelay time.Duration
	// HeaderExtractor is an optional function to extract rate limit info from
	// HTTP response headers. If nil, only internal tracking is used.
	HeaderExtractor HeaderExtractor
}

AdaptiveRateLimitConfig configures the AdaptiveRateLimitProvider.

type AdaptiveRateLimitProvider 

type AdaptiveRateLimitProvider struct {
	// contains filtered or unexported fields
}

AdaptiveRateLimitProvider wraps any Provider with per-provider adaptive rate limiting. It tracks RPM (requests per minute) and TPM (tokens per minute), using a sliding window approach. When the remaining quota drops below a configurable threshold (default 10%), the provider either delays the request until the window resets or returns a clear error.

Rate limit state can be updated from HTTP response headers when a HeaderExtractor is provided, enabling the wrapper to react to server-reported limits even when they differ from configured values.

AdaptiveRateLimitProvider is safe for concurrent use.

Example 
// Create an inner provider (e.g., from NewAnthropicProvider)
var inner Provider = &mockProvider{name: "openai"}

// Wrap with adaptive rate limiting
provider, err := NewAdaptiveRateLimitProvider(inner, AdaptiveRateLimitConfig{
	RPMLimit:         60,    // 60 requests per minute
	TPMLimit:         90000, // 90k tokens per minute
	ThresholdPercent: 10,    // throttle when <10% remaining
	MaxDelay:         5 * time.Second,
	HeaderExtractor:  CommonHeaderExtractor, // parse rate limit headers
})
if err != nil {
	fmt.Printf("Error: %v\n", err)
	return
}

// Use the provider normally
resp, err := provider.Chat(context.Background(), []GraycodeRouterMessage{
	{Role: "user", Content: "Hello!"},
}, ChatOptions{Model: "gpt-4"})
if err != nil {
	fmt.Printf("Error: %v\n", err)
	return
}
fmt.Printf("Response: %s\n", resp.Content)

// Check rate limit status
status := provider.Status()
fmt.Printf("RPM used: %d/%d\n", status.RPMUsed, status.RPMLimit)
fmt.Printf("Tokens used: %d/%d\n", status.TPMUsed, status.TPMLimit)
fmt.Printf("Throttle count: %d\n", status.ThrottleCount)

func NewAdaptiveRateLimitProvider 

func NewAdaptiveRateLimitProvider(inner Provider, config AdaptiveRateLimitConfig) (*AdaptiveRateLimitProvider, error)

NewAdaptiveRateLimitProvider wraps inner with adaptive rate limiting. inner must not be nil (an error is returned otherwise). config may be zero-valued for sensible defaults.

func (*AdaptiveRateLimitProvider) Chat 

func (a *AdaptiveRateLimitProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat sends a non-streaming chat request with adaptive rate limiting.

func (*AdaptiveRateLimitProvider) Name 

func (a *AdaptiveRateLimitProvider) Name() string

Name returns the inner provider name suffixed with "/adaptive-ratelimit".

func (*AdaptiveRateLimitProvider) Ping 

func (a *AdaptiveRateLimitProvider) Ping(ctx context.Context) error

Ping delegates directly to the inner provider.

func (*AdaptiveRateLimitProvider) Status 

func (a *AdaptiveRateLimitProvider) Status() RateLimitState

Status returns a snapshot of the current rate limit state.

func (*AdaptiveRateLimitProvider) StreamChat 

func (a *AdaptiveRateLimitProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat sends a streaming chat request with adaptive rate limiting.

func (*AdaptiveRateLimitProvider) UpdateFromHeaders 

func (a *AdaptiveRateLimitProvider) UpdateFromHeaders(h http.Header)

UpdateFromHeaders updates the rate limit state from HTTP response headers. This can be called externally when headers are available (e.g., from a middleware or response interceptor).

type AgnesClient 

type AgnesClient = adapters.AgnesClient

AgnesClient implements Provider for the Agnes AI API.

func NewAgnesClient 

func NewAgnesClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *AgnesClient

type Alert 

type Alert struct {
	Level     string
	Message   string
	Timestamp time.Time
	Threshold float64
}

Alert represents a usage threshold alert.

type AnthropicCachedMessage 

type AnthropicCachedMessage struct {
	Role         string      `json:"role"`
	Content      interface{} `json:"content"` // string or []CachedContent
	CacheControl interface{} `json:"cache_control,omitempty"`
}

AnthropicCachedMessage is an Anthropic message with optional cache_control.

func AddCacheBreakpoints 

func AddCacheBreakpoints(messages []GraycodeRouterMessage) []AnthropicCachedMessage

AddCacheBreakpoints returns a copy of messages with Anthropic cache_control breakpoints applied following the recommended pattern:

  • Breakpoint on the second-to-last message (caches conversation prefix)
  • The last user message is left uncached (always new)

Only applies to messages with role "user" or "assistant". No-op if fewer than 2 messages.

type AnthropicClient 

type AnthropicClient = adapters.AnthropicClient

AnthropicClient implements Provider for the Anthropic Messages API.

func NewAnthropicClient 

func NewAnthropicClient(apiKey, baseURL string, opts ...ClientOption) *AnthropicClient

Adapter constructors.

type AnthropicClientConfig 

type AnthropicClientConfig struct {
	APIKey         string            `json:"-"`
	DefaultHeaders map[string]string `json:"default_headers,omitempty"`
	Timeout        int               `json:"timeout,omitempty"`
	MaxRetries     int               `json:"max_retries,omitempty"`
	Provider       string            `json:"provider,omitempty"`
	BaseURL        string            `json:"base_url,omitempty"`
}

AnthropicClientConfig holds config for creating an Anthropic client.

type AudioClient 

type AudioClient struct {
	// contains filtered or unexported fields
}

AudioClient transcribes audio via an OpenAI-compatible endpoint.

func NewAudioClient 

func NewAudioClient(apiKey, baseURL string) *AudioClient

NewAudioClient creates a transcription client.

func (*AudioClient) Transcribe 

func (c *AudioClient) Transcribe(ctx context.Context, r TranscriptionRequest) (string, error)

Transcribe sends the audio file and returns the transcript text.

type AzureClient 

type AzureClient = adapters.AzureClient

AzureClient implements Provider for the Azure OpenAI API.

func NewAzureClient 

func NewAzureClient(apiKey, endpoint, apiVersion string) *AzureClient

type BatchClient 

type BatchClient struct {
	// contains filtered or unexported fields
}

BatchClient handles Anthropic Message Batches API (50% cost discount).

func NewBatchClient 

func NewBatchClient(apiKey, baseURL string) *BatchClient

NewBatchClient creates a batch client for Anthropic's batch API.

func (*BatchClient) Poll 

func (bc *BatchClient) Poll(ctx context.Context, batchID string) (*BatchResult, error)

Poll checks the status of a batch. Returns the result when complete.

func (*BatchClient) PollRequestResult 

func (bc *BatchClient) PollRequestResult(ctx context.Context, batchID, customID string, opts PollOptions) (*BatchRequestResult, error)

PollRequestResult reconciles one request: waits until the batch is terminal AND the custom_id's row appears, tolerating eventual consistency where the results endpoint lags batch completion. After the batch completes, up to opts.InitialInterval-scaled extra polls are made before surfacing ErrResultNotVisible.

func (*BatchClient) RequestResults 

func (bc *BatchClient) RequestResults(ctx context.Context, batchID string) ([]BatchRequestResult, error)

RequestResults fetches and parses the newline-delimited results document.

func (*BatchClient) Submit 

func (bc *BatchClient) Submit(ctx context.Context, requests []BatchRequest) (string, error)

Submit sends a batch of requests. Returns the batch ID for polling.

func (*BatchClient) WaitUntilDone 

func (bc *BatchClient) WaitUntilDone(ctx context.Context, batchID string, opts PollOptions) (*BatchResult, error)

WaitUntilDone polls the batch until it reaches a terminal state or the timeout elapses. Non-terminal responses keep polling; 429/5xx responses are retried with Retry-After-aware backoff; other non-200s fail fast.

type BatchRequest 

type BatchRequest struct {
	CustomID string                  `json:"custom_id"`
	Messages []GraycodeRouterMessage `json:"messages"`
	Options  ChatOptions             `json:"options"`
}

BatchRequest represents a single request in a batch.

type BatchRequestResult 

type BatchRequestResult struct {
	CustomID string          `json:"custom_id"`
	Result   json.RawMessage `json:"result,omitempty"`
	Error    json.RawMessage `json:"error,omitempty"`
}

BatchRequestResult is one row of the batch results JSONL: the raw provider payload is preserved byte-exact so hosts decode per-provider shapes.

type BatchResponse 

type BatchResponse struct {
	CustomID string                  `json:"custom_id"`
	Response *GraycodeRouterResponse `json:"response,omitempty"`
	Error    string                  `json:"error,omitempty"`
}

BatchResponse represents a single response from a batch.

type BatchResult 

type BatchResult struct {
	ID        string          `json:"id"`
	Status    string          `json:"status"` // "in_progress", "ended", "failed"
	Responses []BatchResponse `json:"responses,omitempty"`
	CreatedAt time.Time       `json:"created_at"`
}

BatchResult holds the overall batch operation result.

type BedrockClient 

type BedrockClient = adapters.BedrockClient

BedrockClient implements Provider for the AWS Bedrock API.

func NewBedrockClient 

func NewBedrockClient(accessKeyID, secretAccessKey, sessionToken, region string) *BedrockClient

type BudgetProvider 

type BudgetProvider struct {
	// contains filtered or unexported fields
}

BudgetProvider wraps a Provider and enforces per-virtual-key budgets. Before each call it estimates cost and rejects the request if the key is over budget; after a successful call it records the actual spend.

Requests without a virtual key (none in context or options) pass through unmetered, preserving existing behavior.

func NewBudgetProvider 

func NewBudgetProvider(inner Provider, store BudgetStore) *BudgetProvider

NewBudgetProvider wraps inner with budget enforcement backed by store.

func (*BudgetProvider) Chat 

func (bp *BudgetProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat enforces the budget for the request's virtual key, calls the inner provider, then records actual spend.

func (*BudgetProvider) Name 

func (bp *BudgetProvider) Name() string

Name returns the inner provider's name.

func (*BudgetProvider) Ping 

func (bp *BudgetProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*BudgetProvider) StreamChat 

func (bp *BudgetProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat enforces budget up-front, then streams. Actual streamed usage is recorded from the final usage event if present.

type BudgetStore 

type BudgetStore interface {
	// CheckBudget returns ErrBudgetExceeded if charging estCostUSD to the key
	// would exceed its budget, ErrUnknownVirtualKey if the key is unknown, or
	// nil if the request may proceed.
	CheckBudget(ctx context.Context, virtualKey string, estCostUSD float64) error
	// RecordUsage records actual spend against a virtual key after a call.
	RecordUsage(ctx context.Context, virtualKey string, costUSD float64, tokensIn, tokensOut int) error
}

BudgetStore is the persistence contract the BudgetProvider depends on. Its methods use only primitive types so both the in-memory store here and the SQLite store in the storage package satisfy it without an import cycle.

type CacheAnalytics 

type CacheAnalytics struct {
	// contains filtered or unexported fields
}

CacheAnalytics tracks prompt caching effectiveness and cost savings. Shows developers exactly how much money and latency caching saves.

func NewCacheAnalytics 

func NewCacheAnalytics() *CacheAnalytics

NewCacheAnalytics creates a cache analytics tracker.

func (*CacheAnalytics) FormatSummary 

func (ca *CacheAnalytics) FormatSummary() string

FormatSummary returns a human-readable summary.

func (*CacheAnalytics) RecordCall 

func (ca *CacheAnalytics) RecordCall(m CallMetrics)

RecordCall records a call's cache usage from its metrics.

func (*CacheAnalytics) Report 

func (ca *CacheAnalytics) Report() CacheReport

Report returns the current cache effectiveness report.

type CacheConfig 

type CacheConfig struct {
	// MaxAge is how long cache entries remain valid. Default: 5 minutes.
	MaxAge time.Duration
	// MaxSize is the maximum number of cached responses. Default: 100.
	// When exceeded, the least-recently-used entry is evicted.
	MaxSize int
	// Enabled toggles caching. Default: true.
	// When false, the CachedProvider passes all requests through unchanged.
	Enabled bool
	// TemperatureThreshold is the temperature above which responses are not cached.
	// Default: 0.5. Responses with temperature > threshold are expected to vary,
	// so caching them would defeat the purpose.
	TemperatureThreshold float64
}

CacheConfig controls the behavior of CachedProvider.

func DefaultCacheConfig 

func DefaultCacheConfig() CacheConfig

DefaultCacheConfig returns a CacheConfig with sensible defaults.

type CacheControlType 

type CacheControlType string

CacheControlType is the type of cache control. 

const (
	// CacheControlEphemeral caches content for up to 5 minutes.
	CacheControlEphemeral CacheControlType = "ephemeral"
)

type CacheReport 

type CacheReport struct {
	TotalCalls   int           `json:"total_calls"`
	CacheHits    int           `json:"cache_hits"`
	CacheMisses  int           `json:"cache_misses"`
	HitRate      float64       `json:"hit_rate"`
	TokensSaved  int           `json:"tokens_saved"`
	CostSaved    float64       `json:"cost_saved_usd"`
	LatencySaved time.Duration `json:"latency_saved"`
}

CacheReport summarizes caching effectiveness.

type CacheStatsResult 

type CacheStatsResult struct {
	Size    int  `json:"size"`
	MaxSize int  `json:"max_size"`
	Enabled bool `json:"enabled"`
}

CacheStatsResult holds cache statistics.

type CachedContent 

type CachedContent struct {
	Type         string      `json:"type"`
	Text         string      `json:"text"`
	CacheControl interface{} `json:"cache_control,omitempty"`
}

CachedContent wraps a content string with cache control metadata.

type CachedProvider 

type CachedProvider struct {
	// contains filtered or unexported fields
}

CachedProvider wraps a Provider and caches non-streaming responses based on a hash of the input parameters. Inspired by maximhq/bifrost's caching layer.

CachedProvider is safe for concurrent use.

func NewCachedProvider 

func NewCachedProvider(inner Provider, cfg CacheConfig) *CachedProvider

NewCachedProvider wraps inner with a response cache configured by cfg. Zero-value fields in cfg are replaced with defaults.

func (*CachedProvider) CacheStats 

func (cp *CachedProvider) CacheStats() CacheStatsResult

CacheStats returns the current number of entries in the cache.

func (*CachedProvider) Chat 

func (cp *CachedProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat checks the cache first. On a miss, it calls the inner provider and caches the response (if the temperature is not too high).

func (*CachedProvider) ClearCache 

func (cp *CachedProvider) ClearCache()

ClearCache removes all cached entries.

func (*CachedProvider) Name 

func (cp *CachedProvider) Name() string

Name returns the inner provider's name.

func (*CachedProvider) Ping 

func (cp *CachedProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider (no caching).

func (*CachedProvider) SetEnabled 

func (cp *CachedProvider) SetEnabled(enabled bool)

SetEnabled toggles caching at runtime.

func (*CachedProvider) StreamChat 

func (cp *CachedProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat delegates to the inner provider without caching. Streaming responses are inherently incremental and not suitable for simple response caching.

type CallMetrics 

type CallMetrics struct {
	Model               string    `json:"model"`
	Provider            string    `json:"provider"`
	InputTokens         int       `json:"input_tokens"`
	OutputTokens        int       `json:"output_tokens"`
	CacheReadTokens     int       `json:"cache_read_tokens"`
	CacheCreationTokens int       `json:"cache_creation_tokens"`
	LatencyMs           int64     `json:"latency_ms"`
	Timestamp           time.Time `json:"timestamp"`
}

CallMetrics records telemetry for a single LLM API call.

type CallbackProvider 

type CallbackProvider struct {
	// contains filtered or unexported fields
}

CallbackProvider wraps any Provider and invokes registered ProviderCallback hooks at the appropriate points in the request lifecycle.

Callbacks are executed in separate goroutines so they never block the main request path. A panic in a callback is recovered and logged; it does not crash the caller.

CallbackProvider is safe for concurrent use.

func NewCallbackProvider 

func NewCallbackProvider(inner Provider) (*CallbackProvider, error)

NewCallbackProvider wraps the given provider with callback support. The inner provider must not be nil; an error is returned otherwise.

func (*CallbackProvider) AddCallback 

func (cp *CallbackProvider) AddCallback(cb ProviderCallback)

AddCallback registers a callback. It is safe to call from any goroutine.

func (*CallbackProvider) Callbacks 

func (cp *CallbackProvider) Callbacks() []ProviderCallback

Callbacks returns a snapshot of the currently registered callbacks.

func (*CallbackProvider) Chat 

func (cp *CallbackProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat sends a non-streaming chat request. OnRequest is called before the request; OnResponse or OnError is called after.

func (*CallbackProvider) Inner 

func (cp *CallbackProvider) Inner() Provider

Inner returns the wrapped provider.

func (*CallbackProvider) Name 

func (cp *CallbackProvider) Name() string

Name returns the inner provider name suffixed with "/callbacks".

func (*CallbackProvider) Ping 

func (cp *CallbackProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*CallbackProvider) RemoveCallback 

func (cp *CallbackProvider) RemoveCallback(cb ProviderCallback) bool

RemoveCallback removes a previously registered callback by identity (pointer comparison). Returns true if the callback was found and removed.

func (*CallbackProvider) SetLogger 

func (cp *CallbackProvider) SetLogger(l *slog.Logger)

SetLogger sets the logger used for panic-recovery messages.

func (*CallbackProvider) StreamChat 

func (cp *CallbackProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat sends a streaming chat request. OnRequest is called before the request. OnError is called if the initial request fails. OnStreamEvent is called for each event in the resulting stream.

type CanopyWaveClient 

type CanopyWaveClient = adapters.CanopyWaveClient

CanopyWaveClient implements Provider for the CanopyWave API.

func NewCanopyWaveClient 

func NewCanopyWaveClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *CanopyWaveClient

type Cassette 

type Cassette struct {
	Name         string        `json:"name"`
	RecordedAt   time.Time     `json:"recorded_at"`
	Provider     string        `json:"provider"`
	Interactions []Interaction `json:"interactions"`
}

Cassette stores recorded LLM interactions for deterministic replay.

func LoadCassette 

func LoadCassette(path string) (*Cassette, error)

LoadCassette reads a cassette from a JSON file at path.

type ChatOptions 

type ChatOptions = core.ChatOptions

ChatOptions holds options for a chat request.

func ApplyProviderChatDefaults 

func ApplyProviderChatDefaults(provider string, opts ChatOptions) ChatOptions

ApplyProviderChatDefaults applies provider policy that host applications should not need to encode themselves.

type ClientOption 

type ClientOption = core.ClientOption

ClientOption configures clients.

func WithAPIKey 

func WithAPIKey(key string) ClientOption

WithAPIKey sets the API key.

func WithBaseURL 

func WithBaseURL(url string) ClientOption

WithBaseURL sets the base URL.

func WithCoalescing 

func WithCoalescing(ttl time.Duration) ClientOption

WithCoalescing enables request coalescing for identical concurrent requests. When enabled, multiple goroutines sending identical requests (same provider, model, messages, temperature, max_tokens) will be deduplicated into a single API call, with the result broadcast to all waiters.

The ttl parameter controls how long completed requests remain in the coalescer for potential reuse. A typical value is 100-500ms.

func WithGuardrailType 

func WithGuardrailType(types ...GuardrailType) ClientOption

WithGuardrailType attaches output guardrails using built-in rules for the specified types. For example, WithGuardrailType(GuardrailPII, GuardrailSecretLeak) enables PII redaction and secret leak blocking with default patterns.

func WithGuardrails 

func WithGuardrails(rules ...GuardrailRule) ClientOption

WithGuardrails attaches output guardrails to the client. Guardrails run after the LLM response but before returning to the caller. Blocked responses are replaced with an error; redacted responses have matches replaced with asterisks.

func WithHTTPClient 

func WithHTTPClient(hc *http.Client) ClientOption

WithHTTPClient sets a custom HTTP client.

func WithLogger 

func WithLogger(l *slog.Logger) ClientOption

WithLogger sets the logger.

func WithMaxTokens 

func WithMaxTokens(n int) ClientOption

WithMaxTokens sets the default max tokens for requests.

func WithMimoAuth 

func WithMimoAuth() ClientOption

WithMimoAuth uses api-key header per MiMo documentation (OpenAI + Anthropic compat).

func WithModel 

func WithModel(model string) ClientOption

WithModel sets the default model for requests.

func WithProviderName 

func WithProviderName(name string) ClientOption

WithProviderName sets the OpenAI client provider name for errors/logging. No-op for the Anthropic adapter, which reports a fixed provider name.

func WithRetry 

func WithRetry(rc RetryConfig) ClientOption

WithRetry sets retry configuration.

func WithStructuredOutput 

func WithStructuredOutput(schema map[string]interface{}, maxRetries int) ClientOption

WithStructuredOutput returns a ClientOption for structured JSON output.

The option itself is inert: neither adapter is mutated at construction time. Anthropic uses the prefill technique and OpenAI sets response_format, both handled per-call in ChatWithStructuredOutput (which receives the schema through its SchemaValidation parameter — the fields this option previously carried were never read). Kept for API compatibility.

func WithTemperature 

func WithTemperature(t float64) ClientOption

WithTemperature sets the default temperature for requests.

func WithTimeout 

func WithTimeout(d time.Duration) ClientOption

WithTimeout sets the HTTP client timeout.

type ClinePassClient 

type ClinePassClient = adapters.ClinePassClient

ClinePassClient implements Provider for the ClinePass API.

func NewClinePassClient 

func NewClinePassClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *ClinePassClient

type CoalesceKey 

type CoalesceKey struct {
	Provider    string                  `json:"provider"`
	Model       string                  `json:"model"`
	Messages    []GraycodeRouterMessage `json:"messages"`
	Temperature *float64                `json:"temperature,omitempty"`
	MaxTokens   int                     `json:"max_tokens,omitempty"`
}

CoalesceKey uniquely identifies an LLM request for deduplication. It hashes provider, model, messages, temperature, and max_tokens.

func (CoalesceKey) String 

func (k CoalesceKey) String() string

String returns a stable hash of the coalesce key for map lookup.

type Coalescer 

type Coalescer struct {
	// contains filtered or unexported fields
}

Coalescer deduplicates identical concurrent LLM requests. It maintains a map of inflight requests indexed by CoalesceKey. When a request arrives that matches an existing inflight request, the caller waits on the existing request's done channel instead of making a new API call.

The Coalescer automatically cleans up completed requests after a TTL. Coalescer is safe for concurrent use.

func NewCoalescer 

func NewCoalescer(ttl time.Duration) *Coalescer

NewCoalescer creates a new Coalescer with the specified TTL for completed requests. The ttl controls how long completed requests remain in the map for potential reuse.

func (*Coalescer) Coalesce 

func (c *Coalescer) Coalesce(ctx context.Context, key CoalesceKey, fn func() (*GraycodeRouterResponse, error)) (*GraycodeRouterResponse, error)

Coalesce deduplicates concurrent identical requests.

If an inflight request exists for the given key, the caller waits on its done channel and returns the same result. Otherwise, a new InflightRequest is created, added to the inflight map, and fn is called to execute the actual request.

The executing goroutine is responsible for:

  1. Calling fn() to get the result
  2. Storing the result in the InflightRequest
  3. Closing the done channel to wake all waiters

All waiting goroutines receive the same response.

func (*Coalescer) Stats 

func (c *Coalescer) Stats() InflightStats

Stats returns the number of inflight requests (for monitoring/testing).

type ConcentrateResponsesClient 

type ConcentrateResponsesClient = adapters.ConcentrateResponsesClient

ConcentrateResponsesClient implements Provider for the Concentrate Responses API.

func NewConcentrateResponsesClient 

func NewConcentrateResponsesClient(apiKey, baseURL string, opts ...ClientOption) *ConcentrateResponsesClient

type CondenseOptions 

type CondenseOptions struct {
	// MaxSize is the message-count threshold above which condensation runs.
	// When len(messages) <= MaxSize, the history is returned unchanged.
	MaxSize int
	// KeepFirst is the number of leading messages preserved verbatim (e.g. a
	// system prompt and the opening turns). The middle span between the kept
	// head and tail is what gets summarized.
	KeepFirst int
}

CondenseOptions controls how a ConversationCondenser reduces a message history.

type CondenserOption 

type CondenserOption func(*LLMSummarizingCondenser)

CondenserOption configures an LLMSummarizingCondenser.

func WithCondenserMaxTokens 

func WithCondenserMaxTokens(n int) CondenserOption

WithCondenserMaxTokens caps the number of tokens requested for the summary.

func WithCondenserPrompt 

func WithCondenserPrompt(prompt string) CondenserOption

WithCondenserPrompt overrides the default summarization instruction.

func WithCondenserRoles 

func WithCondenserRoles(roles ModelRoles) CondenserOption

WithCondenserRoles sets the model roles used by the condenser. The Weak role is preferred for the summary call.

type CondensingProvider 

type CondensingProvider struct {
	// contains filtered or unexported fields
}

CondensingProvider wraps a Provider and runs a ConversationCondenser over the request messages before delegating to the inner provider. It follows the same decorator pattern as BudgetProvider and TracingProvider.

Condensation applies to both Chat and StreamChat. A nil condenser or non-positive CondenseOptions.MaxSize disables condensation (pass-through).

func NewCondensingProvider 

func NewCondensingProvider(inner Provider, condenser ConversationCondenser, opts CondenseOptions) *CondensingProvider

NewCondensingProvider wraps inner so that request histories are condensed via condenser using the given options. The inner provider must not be nil.

func (*CondensingProvider) Chat 

func (p *CondensingProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat condenses the messages, then delegates to the inner provider.

func (*CondensingProvider) Name 

func (p *CondensingProvider) Name() string

Name returns the inner provider's name.

func (*CondensingProvider) Ping 

func (p *CondensingProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*CondensingProvider) StreamChat 

func (p *CondensingProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat condenses the messages, then delegates to the inner provider.

type ContentPart 

type ContentPart = core.ContentPart

ContentPart represents a piece of content in a multi-modal message.

type ContinuationConfig 

type ContinuationConfig = core.ContinuationConfig

ContinuationConfig controls output continuation behavior.

func DefaultContinuationConfig 

func DefaultContinuationConfig() ContinuationConfig

DefaultContinuationConfig returns sensible defaults.

type ConversationCondenser 

type ConversationCondenser interface {
	// Condense returns a reduced copy of messages according to opts. It must
	// not mutate the input slice. When no reduction is needed it may return the
	// input slice unchanged.
	Condense(ctx context.Context, messages []GraycodeRouterMessage, opts CondenseOptions) ([]GraycodeRouterMessage, error)
}

ConversationCondenser reduces a message history so that long conversations stay within a model's context window while preserving salient information.

type CostEstimate 

type CostEstimate struct {
	InputTokens   int     `json:"input_tokens"`
	OutputTokens  int     `json:"estimated_output_tokens"`
	InputCostUSD  float64 `json:"input_cost_usd"`
	OutputCostUSD float64 `json:"estimated_output_cost_usd"`
	TotalCostUSD  float64 `json:"estimated_total_cost_usd"`
	Model         string  `json:"model"`
	CacheDiscount float64 `json:"cache_discount_usd"` // potential savings if cached
}

CostEstimate is the pre-call cost prediction.

type CostEstimator 

type CostEstimator struct{}

CostEstimator estimates the cost of an API call BEFORE sending it. Helps developers set budgets and avoid surprise charges.

func NewCostEstimator 

func NewCostEstimator() *CostEstimator

NewCostEstimator creates an estimator.

func (*CostEstimator) Estimate 

func (ce *CostEstimator) Estimate(messages []GraycodeRouterMessage, model string, maxOutputTokens int) CostEstimate

Estimate predicts cost for a set of messages + expected output.

func (*CostEstimator) FormatEstimate 

func (ce *CostEstimator) FormatEstimate(est CostEstimate) string

FormatEstimate returns a human-readable cost estimate.

func (*CostEstimator) IsExpensive 

func (ce *CostEstimator) IsExpensive(est CostEstimate, threshold float64) bool

IsExpensive returns true if the estimated cost exceeds a threshold.

type DeepSeekClient 

type DeepSeekClient = adapters.DeepSeekClient

DeepSeekClient implements Provider for the DeepSeek API.

func NewDeepSeekClient 

func NewDeepSeekClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *DeepSeekClient

type DeprecationChecker 

type DeprecationChecker struct {
	// contains filtered or unexported fields
}

DeprecationChecker warns when using models approaching retirement. Read-only after construction — no mutex needed.

func NewDeprecationChecker 

func NewDeprecationChecker() *DeprecationChecker

NewDeprecationChecker creates a checker with known deprecations.

func (*DeprecationChecker) Check 

func (dc *DeprecationChecker) Check(model string) *DeprecationInfo

Check returns deprecation info if the model is deprecated.

func (*DeprecationChecker) Warn 

func (dc *DeprecationChecker) Warn(model string)

Warn logs a deprecation warning if applicable.

type DeprecationInfo 

type DeprecationInfo struct {
	Model       string    `json:"model"`
	Deprecated  bool      `json:"deprecated"`
	RetireDate  time.Time `json:"retire_date"`
	Replacement string    `json:"replacement"`
	Message     string    `json:"message"`
}

DeprecationInfo describes a model's deprecation status.

type Embedder 

type Embedder = embeddings.Embedder

Embedder is the interface for creating embeddings.

type EmbeddingCachedProvider 

type EmbeddingCachedProvider = embeddings.EmbeddingCachedProvider

EmbeddingCachedProvider caches chat responses keyed by embedding similarity.

func NewEmbeddingCachedProvider 

func NewEmbeddingCachedProvider(inner Provider, embedder Embedder, cfg SemanticCacheConfig) *EmbeddingCachedProvider

NewEmbeddingCachedProvider wraps a provider with an embedding-similarity cache.

type EmbeddingParams 

type EmbeddingParams = embeddings.EmbeddingParams

EmbeddingParams holds asymmetric params for indexing vs query.

func DefaultEmbeddingParams 

func DefaultEmbeddingParams(model string) EmbeddingParams

DefaultEmbeddingParams returns known-good asymmetric params for common embedding models.

type EmbeddingRequest 

type EmbeddingRequest = embeddings.EmbeddingRequest

EmbeddingRequest represents an embedding API call.

type EmbeddingResponse 

type EmbeddingResponse = embeddings.EmbeddingResponse

EmbeddingResponse holds embedding results.

type ExtractOptions 

type ExtractOptions struct {
	// Chat carries provider/model/temperature for the extraction call. If Model
	// is empty the provider default is used.
	Chat ChatOptions
	// Instruction overrides the default extraction instruction. Use it to scope
	// what relations to extract (e.g. "extract only code dependency relations").
	// When empty, a general noun-constrained instruction is used.
	Instruction string
	// AllowedPredicates, when non-empty, constrains the predicate vocabulary —
	// the model is told to use only these relation types, and triples with other
	// predicates are dropped after extraction. This is the lightweight analogue
	// of CocoIndex's EntityTypeConfig schema constraint.
	AllowedPredicates []string
	// MaxRetries is the schema-validation retry budget. Defaults to 2.
	MaxRetries int
}

ExtractOptions configures triple extraction. The zero value is valid and uses sensible defaults (noun-constrained entities, 2 validation retries).

type FeatureSet 

type FeatureSet struct {
	Thinking         bool `json:"thinking"`
	AdaptiveThinking bool `json:"adaptive_thinking"`
	ToolUse          bool `json:"tool_use"`
	Images           bool `json:"images"`
	Streaming        bool `json:"streaming"`
	Caching          bool `json:"caching"`
	JSON             bool `json:"json_mode"`
	Embeddings       bool `json:"embeddings"`
	MaxContext       int  `json:"max_context"`
	MaxOutput        int  `json:"max_output"`
	Effort           bool `json:"effort"`
	StructuredOutput bool `json:"structured_output"`
	CodeExecution    bool `json:"code_execution"`
	Citations        bool `json:"citations"`
	PDFInput         bool `json:"pdf_input"`
}

FeatureSet describes what a provider or model supports. When the catalog is loaded, per-model values from the live API take precedence over the hardcoded defaults.

type GeminiClient 

type GeminiClient = adapters.GeminiClient

GeminiClient implements Provider for the Google Gemini API.

func NewGeminiClient 

func NewGeminiClient(apiKey, baseURL string) *GeminiClient

type GeminiOpenAIClient 

type GeminiOpenAIClient = adapters.GeminiOpenAIClient

GeminiOpenAIClient implements Provider for the OpenAI-compatible Gemini endpoint.

func NewGeminiOpenAIClient 

func NewGeminiOpenAIClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *GeminiOpenAIClient

type GraycodeRouterClient 

type GraycodeRouterClient struct {
	// contains filtered or unexported fields
}

GraycodeRouterClient is the universal LLM client. It is safe for concurrent use.

func Client 

func Client(cfg *GraycodeRouterConfig, opts ...ClientOption) *GraycodeRouterClient

Client creates an GraycodeRouterClient.

func (*GraycodeRouterClient) Chat 

func (c *GraycodeRouterClient) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat sends a chat request to the specified (or default) provider.

func (*GraycodeRouterClient) ChatWithStructuredOutput 

func (c *GraycodeRouterClient) ChatWithStructuredOutput(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions, validation SchemaValidation) (*GraycodeRouterResponse, error)

ChatWithStructuredOutput sends a chat request with structured output validation. If the response doesn't match the schema, it retries with error feedback.

func (*GraycodeRouterClient) CreateEmbedding 

func (c *GraycodeRouterClient) CreateEmbedding(ctx context.Context, req EmbeddingRequest, provider string) (*EmbeddingResponse, error)

CreateEmbedding sends an embedding request to the specified (or default) provider.

func (*GraycodeRouterClient) ExtractRelationships 

func (c *GraycodeRouterClient) ExtractRelationships(ctx context.Context, text string, opts ExtractOptions) ([]Relationship, error)

ExtractRelationships extracts subject-predicate-object triples from text using schema-validated structured output with retry. It is a typed convenience layer over ChatWithStructuredOutput, modeled on CocoIndex's ExtractByLlm; Harrier and other knowledge-graph consumers can call it instead of hand-rolling extraction prompts and JSON parsing.

func (*GraycodeRouterClient) GetProviderInfo 

func (c *GraycodeRouterClient) GetProviderInfo(provider string) *ProviderRegistryConfig

GetProviderInfo returns config for a provider.

func (*GraycodeRouterClient) GetProviders 

func (c *GraycodeRouterClient) GetProviders() []string

GetProviders lists all available providers.

func (*GraycodeRouterClient) Ping 

func (c *GraycodeRouterClient) Ping(ctx context.Context, provider string) error

Ping checks connectivity to the specified (or default) provider.

func (*GraycodeRouterClient) SetAPIKey 

func (c *GraycodeRouterClient) SetAPIKey(provider, apiKey string)

SetAPIKey sets an API key for a provider.

func (*GraycodeRouterClient) SetCoalescingTTL 

func (c *GraycodeRouterClient) SetCoalescingTTL(ttl time.Duration)

SetCoalescingTTL enables request coalescing with the given reuse TTL. Implements core.GraycodeRouterConfigurable for WithCoalescing.

func (*GraycodeRouterClient) StreamChat 

func (c *GraycodeRouterClient) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat sends a streaming chat request.

func (*GraycodeRouterClient) StreamChatContinue 

func (c *GraycodeRouterClient) StreamChatContinue(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions, cfg ContinuationConfig) (*StreamResult, error)

StreamChatContinue is like StreamChat but automatically continues if the response hits max_tokens with text-only content. Continuations are transparent to the caller.

type GraycodeRouterConfig 

type GraycodeRouterConfig = core.GraycodeRouterConfig

GraycodeRouterConfig holds client configuration.

type GraycodeRouterError 

type GraycodeRouterError = core.GraycodeRouterError

GraycodeRouterError is a structured error that preserves provider context, HTTP metadata, and request identification for debugging.

type GraycodeRouterMessage 

type GraycodeRouterMessage = core.GraycodeRouterMessage

GraycodeRouterMessage represents a chat message.

func BuildStructuredPrompt 

func BuildStructuredPrompt(messages []GraycodeRouterMessage, schema map[string]interface{}) []GraycodeRouterMessage

BuildStructuredPrompt adds JSON schema instructions to the message system prompt. It prepends schema requirements to ensure the LLM outputs valid JSON matching the schema.

func MergeConsecutiveRoles 

func MergeConsecutiveRoles(messages []GraycodeRouterMessage) []GraycodeRouterMessage

MergeConsecutiveRoles merges adjacent messages that share the same role by concatenating their content with a newline separator.

Messages with ToolUse or ToolResult are never merged, since those have special provider semantics and must remain separate.

func NewAudioMessage 

func NewAudioMessage(base64Data, format string) GraycodeRouterMessage

NewAudioMessage creates a user message with base64-encoded audio. format should be "wav" or "mp3".

func NewAudioMessageWithText 

func NewAudioMessageWithText(text, base64Data, format string) GraycodeRouterMessage

NewAudioMessageWithText creates a user message with text and base64-encoded audio.

func NewBase64ImageMessage 

func NewBase64ImageMessage(data, mediaType string) GraycodeRouterMessage

NewBase64ImageMessage creates a user message with a base64-encoded image. mediaType should be a MIME type like "image/png" or "image/jpeg".

func NewBase64ImageMessageWithText 

func NewBase64ImageMessageWithText(text, data, mediaType string) GraycodeRouterMessage

NewBase64ImageMessageWithText creates a user message with text and a base64-encoded image.

func NewImageMessage 

func NewImageMessage(url string) GraycodeRouterMessage

NewImageMessage creates a user message with an image from a URL or data URI. The url parameter accepts HTTP(S) URLs or data URIs (data:image/png;base64,...).

func NewImageMessageWithText 

func NewImageMessageWithText(text, url string) GraycodeRouterMessage

NewImageMessageWithText creates a user message with text and an image from a URL or data URI.

func SanitizeMessages 

func SanitizeMessages(messages []GraycodeRouterMessage) []GraycodeRouterMessage

SanitizeMessages inspects messages for orphaned tool_use blocks and injects synthetic error results. Implementation lives in client/core.

type GraycodeRouterResponse 

type GraycodeRouterResponse = core.GraycodeRouterResponse

GraycodeRouterResponse is the response from a chat call.

func ChatWithContinuation 

func ChatWithContinuation(ctx context.Context, p Provider, messages []GraycodeRouterMessage, opts ChatOptions, cfg ContinuationConfig) (*GraycodeRouterResponse, error)

ChatWithContinuation calls Chat and automatically continues if stop_reason is "max_tokens". It appends the partial response as an assistant message and retries, accumulating content. Returns the fully assembled response.

type GraycodeRouterStreamEvent 

type GraycodeRouterStreamEvent = core.GraycodeRouterStreamEvent

GraycodeRouterStreamEvent is a streaming event.

type GraycodeRouterTool 

type GraycodeRouterTool = core.GraycodeRouterTool

GraycodeRouterTool represents a tool definition.

type GraycodeRouterUsage 

type GraycodeRouterUsage = core.GraycodeRouterUsage

GraycodeRouterUsage tracks token usage.

type GrokClient 

type GrokClient = adapters.GrokClient

GrokClient implements Provider for the xAI (Grok) API.

func NewGrokClient 

func NewGrokClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *GrokClient

type GroqClient 

type GroqClient = adapters.GroqClient

GroqClient implements Provider for the Groq API.

func NewGroqClient 

func NewGroqClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *GroqClient

type GuardrailAction 

type GuardrailAction = core.GuardrailAction

GuardrailAction is what happens when a guardrail rule matches.

type GuardrailError 

type GuardrailError = core.GuardrailError

GuardrailError is returned when a blocking rule matches.

type GuardrailProvider 

type GuardrailProvider struct {
	// contains filtered or unexported fields
}

GuardrailProvider wraps any Provider and runs guardrail checks on LLM responses before returning them to the caller. When a guardrail with Action=Block matches, the response is replaced with an error. When Action=Redact, matched content is scrubbed. When Action=Warn, the violation is logged but the response passes through unchanged.

GuardrailProvider is safe for concurrent use.

func NewGuardrailProvider 

func NewGuardrailProvider(inner Provider, g *Guardrails) *GuardrailProvider

NewGuardrailProvider wraps the given provider with output guardrails. The inner provider must not be nil. The guardrails parameter may be nil (in which case the wrapper is a no-op).

func (*GuardrailProvider) Chat 

func (gp *GuardrailProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat sends a chat request and validates the response against guardrails.

func (*GuardrailProvider) Inner 

func (gp *GuardrailProvider) Inner() Provider

Inner returns the wrapped provider.

func (*GuardrailProvider) Name 

func (gp *GuardrailProvider) Name() string

Name returns the inner provider name suffixed with "/guardrails".

func (*GuardrailProvider) Ping 

func (gp *GuardrailProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*GuardrailProvider) StreamChat 

func (gp *GuardrailProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat sends a streaming request and validates content events. Blocked violations cause the stream to be cancelled and an error event emitted. Redactions are applied to individual content chunks.

type GuardrailRule 

type GuardrailRule = core.GuardrailRule

GuardrailRule is one output-filtering rule.

func AllDefaultRules 

func AllDefaultRules() []GuardrailRule

AllDefaultRules returns every built-in guardrail rule.

func DefaultHarmfulContentRules 

func DefaultHarmfulContentRules() []GuardrailRule

DefaultHarmfulContentRules returns the built-in harmful-content rules.

func DefaultPIIRules 

func DefaultPIIRules() []GuardrailRule

DefaultPIIRules returns the built-in PII redaction rules.

func DefaultPromptInjectionRules 

func DefaultPromptInjectionRules() []GuardrailRule

DefaultPromptInjectionRules returns the built-in prompt-injection rules.

func DefaultSecretLeakRules 

func DefaultSecretLeakRules() []GuardrailRule

DefaultSecretLeakRules returns the built-in secret-leak blocking rules.

func RulesForType 

func RulesForType(t GuardrailType) []GuardrailRule

RulesForType returns the built-in rules for one guardrail type.

type GuardrailSeverity 

type GuardrailSeverity = core.GuardrailSeverity

GuardrailSeverity ranks how serious a violation is.

type GuardrailType 

type GuardrailType = core.GuardrailType

GuardrailType classifies a guardrail rule.

type GuardrailViolation 

type GuardrailViolation = core.GuardrailViolation

GuardrailViolation records a rule match in a response.

type Guardrails 

type Guardrails = core.Guardrails

Guardrails is a compiled set of output-filtering rules.

func NewGuardrails 

func NewGuardrails(rules ...GuardrailRule) *Guardrails

NewGuardrails compiles a guardrail rule set (panics on invalid patterns).

func NewGuardrailsSafe 

func NewGuardrailsSafe(rules ...GuardrailRule) (*Guardrails, error)

NewGuardrailsSafe compiles a guardrail rule set, returning pattern errors.

type HeaderExtractor 

type HeaderExtractor func(h http.Header) *RateLimitHeaders

HeaderExtractor is a function that extracts rate limit information from HTTP response headers. Different providers use different header naming conventions (OpenAI uses x-ratelimit-*, Anthropic uses anthropic-ratelimit-*).

type ImageClient 

type ImageClient struct {
	// contains filtered or unexported fields
}

ImageClient generates images via an OpenAI-compatible endpoint.

func NewImageClient 

func NewImageClient(apiKey, baseURL string) *ImageClient

NewImageClient creates an image client. baseURL defaults to https://api.openai.com; set it to an OpenAI-compatible endpoint for others.

func (*ImageClient) Generate 

func (c *ImageClient) Generate(ctx context.Context, prompt, model, size string, n int) ([][]byte, []string, error)

Generate creates n images for prompt. Returns each as bytes (b64 decoded) plus the provider URL when present. size is e.g. "1024x1024".

type ImageGenRequest 

type ImageGenRequest struct {
	Model          string `json:"model"`
	Prompt         string `json:"prompt"`
	N              int    `json:"n,omitempty"`
	Size           string `json:"size,omitempty"`
	ResponseFormat string `json:"response_format,omitempty"` // "url" | "b64_json"
}

ImageGenRequest is the body for POST /v1/images/generations.

type ImageGenResponse 

type ImageGenResponse struct {
	Created int64            `json:"created"`
	Data    []ImageGenResult `json:"data"`
}

ImageGenResponse is the top-level response.

type ImageGenResult 

type ImageGenResult struct {
	URL           string `json:"url,omitempty"`
	B64JSON       string `json:"b64_json,omitempty"`
	RevisedPrompt string `json:"revised_prompt,omitempty"`
}

ImageGenResult is one generated image.

type ImageURLPart 

type ImageURLPart = core.ImageURLPart

ImageURLPart represents an image content part.

type InflightRequest 

type InflightRequest struct {
	// contains filtered or unexported fields
}

InflightRequest represents a pending request that multiple waiters can join. The first goroutine to create this request is responsible for executing it and broadcasting the result to all waiting goroutines.

type InflightStats 

type InflightStats struct {
	// InflightRequests is the number of unique requests being executed
	InflightRequests int
	// TotalWaiters is the total number of goroutines waiting across all requests
	TotalWaiters int
}

InflightStats contains statistics about inflight coalesced requests.

type InputAudioPart 

type InputAudioPart = core.InputAudioPart

InputAudioPart represents an audio content part (base64 encoded).

type Interaction 

type Interaction struct {
	Request  RecordedRequest  `json:"request"`
	Response RecordedResponse `json:"response"`
}

Interaction pairs a recorded request with its response.

type KimiClient 

type KimiClient = adapters.KimiClient

KimiClient implements Provider for the Kimi (Moonshot) API.

func NewKimiClient 

func NewKimiClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *KimiClient

type LLMSummarizingCondenser 

type LLMSummarizingCondenser struct {
	// contains filtered or unexported fields
}

LLMSummarizingCondenser condenses a conversation by summarizing its middle span via an LLM call, keeping the first KeepFirst messages and the tail intact. The summary is inserted as a single system note between the head and the tail.

The summary call uses the Weak model role when a ModelRoles is configured (see WithModelRoles / ResolveRole), so summarization runs on a cheaper model.

func NewLLMSummarizingCondenser 

func NewLLMSummarizingCondenser(provider Provider, opts ...CondenserOption) *LLMSummarizingCondenser

NewLLMSummarizingCondenser creates a condenser that summarizes via the given provider. The provider must not be nil.

func (*LLMSummarizingCondenser) Condense 

func (c *LLMSummarizingCondenser) Condense(ctx context.Context, messages []GraycodeRouterMessage, opts CondenseOptions) ([]GraycodeRouterMessage, error)

Condense implements ConversationCondenser. When len(messages) exceeds MaxSize, it keeps the first KeepFirst messages, summarizes the middle span, inserts the summary as a system note, and keeps the remaining tail.

type LazyProvider 

type LazyProvider struct {
	// contains filtered or unexported fields
}

LazyProvider adapts GraycodeRouterClient to the Provider interface without eagerly resolving credentials or constructing a concrete provider.

func NewLazyProvider 

func NewLazyProvider(cfg *GraycodeRouterConfig) *LazyProvider

NewLazyProvider creates a provider wrapper that resolves the concrete provider only when chat or ping operations are invoked.

func (*LazyProvider) Chat 

func (p *LazyProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

func (*LazyProvider) Name 

func (p *LazyProvider) Name() string

func (*LazyProvider) Ping 

func (p *LazyProvider) Ping(ctx context.Context) error

func (*LazyProvider) SetAPIKey 

func (p *LazyProvider) SetAPIKey(provider, apiKey string)

func (*LazyProvider) StreamChat 

func (p *LazyProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

type LongCatClient 

type LongCatClient = adapters.LongCatClient

LongCatClient implements Provider for the LongCat API.

func NewLongCatClient 

func NewLongCatClient(apiKey, openAIBase, anthropicBase string, compat *OpenAICompatConfig, opts ...ClientOption) *LongCatClient

type MemoryBudgetStore 

type MemoryBudgetStore struct {
	// contains filtered or unexported fields
}

MemoryBudgetStore is an in-memory BudgetStore suitable for tests and single-process use. It is safe for concurrent use.

func NewMemoryBudgetStore 

func NewMemoryBudgetStore() *MemoryBudgetStore

NewMemoryBudgetStore creates an empty in-memory budget store.

func (*MemoryBudgetStore) CheckBudget 

func (s *MemoryBudgetStore) CheckBudget(_ context.Context, virtualKey string, estCostUSD float64) error

CheckBudget implements BudgetStore.

func (*MemoryBudgetStore) RecordUsage 

func (s *MemoryBudgetStore) RecordUsage(_ context.Context, virtualKey string, costUSD float64, tokensIn, tokensOut int) error

RecordUsage implements BudgetStore.

func (*MemoryBudgetStore) SetBudget 

func (s *MemoryBudgetStore) SetBudget(virtualKey string, limitUSD float64)

SetBudget creates or updates a virtual key with the given USD limit. A non-positive limit means unlimited.

func (*MemoryBudgetStore) Usage 

func (s *MemoryBudgetStore) Usage(virtualKey string) (usedUSD float64, tokensIn, tokensOut int, ok bool)

Usage returns the recorded spend for a virtual key.

type MetricsCollector 

type MetricsCollector struct {
	// contains filtered or unexported fields
}

MetricsCollector stores recent call metrics in a ring buffer.

func NewMetricsCollector 

func NewMetricsCollector() *MetricsCollector

NewMetricsCollector creates a new MetricsCollector.

func (*MetricsCollector) Recent 

func (mc *MetricsCollector) Recent(n int) []CallMetrics

Recent returns the last n call metrics, most recent first. If fewer than n entries exist, all available entries are returned.

func (*MetricsCollector) Record 

func (mc *MetricsCollector) Record(m CallMetrics)

Record adds a new CallMetrics entry to the ring buffer.

func (*MetricsCollector) TotalCost 

func (mc *MetricsCollector) TotalCost() float64

TotalCost estimates the total cost across all recorded metrics using a simplified pricing model (per 1M tokens):

  • Input tokens: $3.00 / 1M
  • Output tokens: $15.00 / 1M
  • Cache read tokens: $0.30 / 1M
  • Cache creation tokens: $3.75 / 1M

type MiMoClient 

type MiMoClient = adapters.MiMoClient

MiMoClient implements Provider for the Xiaomi MiMo API.

func NewMiMoClient 

func NewMiMoClient(apiKey, openAIBase string, compat *OpenAICompatConfig, providerID string, opts ...ClientOption) *MiMoClient

type MiniMaxClient 

type MiniMaxClient = adapters.MiniMaxClient

MiniMaxClient implements Provider for the MiniMax API.

func NewMiniMaxClient 

func NewMiniMaxClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *MiniMaxClient

type MockCall 

type MockCall struct {
	Messages []GraycodeRouterMessage
	Options  ChatOptions
}

MockCall records a single call to the mock provider.

type MockMode 

type MockMode string

MockMode controls how the mock provider responds. 

const (
	// MockModeEcho echoes the last user message back.
	MockModeEcho MockMode = "echo"
	// MockModeFixed returns a fixed response set via MockProvider.Response.
	MockModeFixed MockMode = "fixed"
	// MockModeToolUse returns a tool call response.
	MockModeToolUse MockMode = "tool_use"
	// MockModeError always returns an error.
	MockModeError MockMode = "error"
	// MockModeMaxTokens returns a response with stop_reason=max_tokens (for testing continuation).
	MockModeMaxTokens MockMode = "max_tokens"
)

type MockProvider 

type MockProvider struct {
	Mode     MockMode
	Response string // used in MockModeFixed
	ToolName string // used in MockModeToolUse
	ToolArgs map[string]interface{}
	Delay    time.Duration // simulate latency
	Calls    []MockCall    // recorded calls for assertions
	// contains filtered or unexported fields
}

MockProvider is a Provider implementation for testing. It never makes real HTTP requests.

func NewMockProvider 

func NewMockProvider(mode MockMode) *MockProvider

NewMockProvider creates a mock provider with the given mode.

func (*MockProvider) CallCount 

func (m *MockProvider) CallCount() int

CallCount returns the number of recorded calls.

func (*MockProvider) Chat 

func (m *MockProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat returns a mock response based on Mode.

func (*MockProvider) LastCall 

func (m *MockProvider) LastCall() *MockCall

LastCall returns the most recent recorded call, or nil.

func (*MockProvider) MarshalCalls 

func (m *MockProvider) MarshalCalls() string

MarshalCalls returns recorded calls as JSON for debugging.

func (*MockProvider) Name 

func (m *MockProvider) Name() string

Name returns "mock".

func (*MockProvider) Ping 

func (m *MockProvider) Ping(_ context.Context) error

Ping always succeeds.

func (*MockProvider) Reset 

func (m *MockProvider) Reset()

Reset clears recorded calls.

func (*MockProvider) StreamChat 

func (m *MockProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat streams a mock response word by word.

type ModelRoles 

type ModelRoles struct {
	Primary string `json:"primary,omitempty"`
	Weak    string `json:"weak,omitempty"`
	Editor  string `json:"editor,omitempty"`
}

ModelRoles maps named role slots to concrete model ids. Empty fields fall back to Primary via ResolveRole.

type ModerationOption 

type ModerationOption func(*ModerationProvider)

ModerationOption configures a ModerationProvider.

func WithBlockedPatterns 

func WithBlockedPatterns(patterns []string) ModerationOption

WithBlockedPatterns sets regex patterns that will block matching input. Each pattern is compiled as a Go regexp.

func WithCustomChecker 

func WithCustomChecker(fn func(string) error) ModerationOption

WithCustomChecker sets a custom validation function that receives the concatenated text content of all messages. If the function returns a non-nil error, the request is blocked.

func WithModerationMaxTokens 

func WithModerationMaxTokens(n int) ModerationOption

WithModerationMaxTokens sets the maximum allowed total token count across all input messages. Token count is estimated as len(strings.Fields(text)) for simplicity. A value of 0 (default) means no limit.

type ModerationProvider 

type ModerationProvider struct {
	// contains filtered or unexported fields
}

ModerationProvider wraps any Provider and validates input messages before forwarding requests. It checks for blocked patterns (regex), token count limits, and custom safety checks.

ModerationProvider is safe for concurrent use.

func NewModerationProvider 

func NewModerationProvider(inner Provider, opts ...ModerationOption) *ModerationProvider

NewModerationProvider wraps inner with content moderation. At least one moderation option should be provided or the wrapper is a no-op.

func (*ModerationProvider) Chat 

func (mp *ModerationProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat validates the input messages and forwards to the inner provider.

func (*ModerationProvider) Name 

func (mp *ModerationProvider) Name() string

Name returns the inner provider name suffixed with "/moderation".

func (*ModerationProvider) Ping 

func (mp *ModerationProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*ModerationProvider) StreamChat 

func (mp *ModerationProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat validates the input messages and forwards to the inner provider.

type OllamaClient 

type OllamaClient = adapters.OllamaClient

OllamaClient implements Provider for the Ollama API.

func NewOllamaClient 

func NewOllamaClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *OllamaClient

type OpenAIClient 

type OpenAIClient = adapters.OpenAIClient

OpenAIClient implements Provider for the OpenAI Chat Completions API.

func NewOpenAIClient 

func NewOpenAIClient(apiKey, baseURL string, compat *OpenAICompatConfig, opts ...ClientOption) *OpenAIClient

type OpenAICompatConfig 

type OpenAICompatConfig = adapters.OpenAICompatConfig

OpenAICompatConfig holds provider-specific compatibility flags.

type OpenCodeGoClient 

type OpenCodeGoClient = adapters.OpenCodeGoClient

OpenCodeGoClient implements Provider for the OpenCode Go API.

func NewOpenCodeGoClient 

func NewOpenCodeGoClient(apiKey, baseURL string, opts ...ClientOption) *OpenCodeGoClient

type OpenGatewayClient 

type OpenGatewayClient = adapters.OpenGatewayClient

OpenGatewayClient implements Provider for the OpenGateway API.

func NewOpenGatewayClient 

func NewOpenGatewayClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *OpenGatewayClient

type OpenRouterClient 

type OpenRouterClient = adapters.OpenRouterClient

OpenRouterClient implements Provider for the OpenRouter API.

func NewOpenRouterClient 

func NewOpenRouterClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *OpenRouterClient

type PollOptions 

type PollOptions struct {
	// InitialInterval is the first sleep between polls (default 2s).
	InitialInterval time.Duration
	// MaxInterval caps the exponential growth of the sleep (default 30s).
	MaxInterval time.Duration
	// Timeout bounds total wall-clock waiting (default 10m).
	Timeout time.Duration
	// JitterFraction randomizes each sleep by ± this fraction (default 0.2).
	JitterFraction float64
}

PollOptions configures wait loops.

type PoolsideClient 

type PoolsideClient = adapters.PoolsideClient

PoolsideClient implements Poolside reasoning-only stream recovery.

func NewPoolsideClient 

func NewPoolsideClient(apiKey, baseURL string, opts ...ClientOption) *PoolsideClient

type PromptOptimizer 

type PromptOptimizer struct {
	// contains filtered or unexported fields
}

PromptOptimizer compresses conversation history to reduce input tokens. Keeps the most recent messages intact but summarizes older ones.

func NewPromptOptimizer 

func NewPromptOptimizer(maxInputTokens int) *PromptOptimizer

NewPromptOptimizer creates an optimizer with a token budget.

func (*PromptOptimizer) Optimize 

func (po *PromptOptimizer) Optimize(messages []GraycodeRouterMessage) []GraycodeRouterMessage

Optimize compresses messages to fit within the token budget. Preserves: system message, first user message, last N messages. Summarizes: middle messages.

type ProtocolRouter 

type ProtocolRouter = adapters.ProtocolRouter

ProtocolRouter routes between OpenAI and Anthropic protocols.

type ProtocolStreamConfig 

type ProtocolStreamConfig = adapters.ProtocolStreamConfig

ProtocolStreamConfig controls streaming across two protocols.

type Provider 

type Provider = core.Provider

Provider is the core interface for LLM providers.

func WithRateLimit 

func WithRateLimit(p Provider, limiter *RateLimiter) Provider

WithRateLimit wraps a provider with a rate limiter.

type ProviderCallback 

type ProviderCallback interface {
	// OnRequest is called before each Chat or StreamChat request.
	// The messages and opts parameters must not be modified.
	OnRequest(ctx context.Context, provider string, model string, messages []GraycodeRouterMessage, opts ChatOptions)

	// OnResponse is called after a successful Chat request.
	OnResponse(ctx context.Context, provider string, model string, response *GraycodeRouterResponse, duration time.Duration)

	// OnError is called after a Chat or StreamChat request fails.
	OnError(ctx context.Context, provider string, model string, err error, duration time.Duration)

	// OnStreamEvent is called for each event emitted during streaming.
	OnStreamEvent(ctx context.Context, provider string, model string, event GraycodeRouterStreamEvent)
}

ProviderCallback defines hooks that are invoked at various points during provider request lifecycle. All methods are optional — implement only the ones you need. Implementations MUST be safe for concurrent use.

type ProviderFeatures 

type ProviderFeatures struct {
	// contains filtered or unexported fields
}

ProviderFeatures tracks which capabilities each provider supports. Prevents sending unsupported features (thinking, tools, images) to providers that don't handle them, avoiding cryptic API errors. Read-only after construction — no mutex needed.

func NewProviderFeatures 

func NewProviderFeatures() *ProviderFeatures

NewProviderFeatures creates a feature registry. The catalog is the single source of truth for per-model capabilities. The hardcoded map is empty — all values come from the live API via the catalog.

func (*ProviderFeatures) Get 

func (pf *ProviderFeatures) Get(provider string) FeatureSet

Get returns features for a provider or model. The compiled catalog (populated from the live API) is the single source of truth. Returns zero-value FeatureSet if the catalog is not loaded — caller must ensure the catalog is loaded before querying features.

func (*ProviderFeatures) Supports 

func (pf *ProviderFeatures) Supports(provider, feature string) bool

Supports checks if a provider supports a specific feature.

type ProviderHealth 

type ProviderHealth struct {
	// contains filtered or unexported fields
}

ProviderHealth tracks latency and error rates per provider. Enables intelligent routing: prefer the healthiest provider.

func NewProviderHealth 

func NewProviderHealth() *ProviderHealth

NewProviderHealth creates a health tracker.

func (*ProviderHealth) AllScores 

func (ph *ProviderHealth) AllScores() []ProviderScore

AllScores returns health scores for all tracked providers.

func (*ProviderHealth) Healthiest 

func (ph *ProviderHealth) Healthiest(candidates []string) string

Healthiest returns the provider with the best health score.

func (*ProviderHealth) RecordFailure 

func (ph *ProviderHealth) RecordFailure(provider string, latency time.Duration)

RecordFailure records a failed API call.

func (*ProviderHealth) RecordSuccess 

func (ph *ProviderHealth) RecordSuccess(provider string, latency time.Duration)

RecordSuccess records a successful API call.

func (*ProviderHealth) Score 

func (ph *ProviderHealth) Score(provider string) float64

Score returns health score for a provider (0-1).

type ProviderRegistryConfig 

type ProviderRegistryConfig = adapters.ProviderRegistryConfig

ProviderRegistryConfig holds provider registry info.

type ProviderScore 

type ProviderScore struct {
	Name         string  `json:"name"`
	Score        float64 `json:"score"` // 0-1 (1 = perfect health)
	AvgLatencyMs int64   `json:"avg_latency_ms"`
	ErrorRate    float64 `json:"error_rate"`
	IsHealthy    bool    `json:"is_healthy"`
}

ProviderScore represents a provider's health score.

type ProviderType 

type ProviderType = adapters.ProviderType

ProviderType classifies providers.

type RateLimitConfig 

type RateLimitConfig struct {
	// RequestsPerMinute is the maximum requests per minute (0 = unlimited).
	RequestsPerMinute int
	// BurstSize is the maximum burst above the steady rate (default = RequestsPerMinute/10, min 1).
	BurstSize int
	// MinInterval is the minimum time between requests (e.g., 5ms for cloud, 0 for local).
	MinInterval time.Duration
}

RateLimitConfig holds rate limit settings for a provider.

type RateLimitHeaders 

type RateLimitHeaders struct {
	RequestsRemaining int
	RequestsLimit     int
	TokensRemaining   int
	TokensLimit       int
	ResetTime         time.Time
}

RateLimitHeaders contains rate limit information extracted from HTTP response headers. Different providers use different header names; common patterns include OpenAI's x-ratelimit-* and Anthropic's anthropic-ratelimit-* headers.

func CommonHeaderExtractor 

func CommonHeaderExtractor(h http.Header) *RateLimitHeaders

CommonHeaderExtractor tries to parse rate limit headers from common LLM API providers (OpenAI, Anthropic, and compatible APIs).

type RateLimitState 

type RateLimitState struct {
	// RPM tracking
	RPMUsed    int       // requests used in the current window
	RPMLimit   int       // maximum requests per minute (0 = unknown)
	RPMResetAt time.Time // when the RPM window resets

	// TPM tracking
	TPMUsed    int       // tokens used in the current window
	TPMLimit   int       // maximum tokens per minute (0 = unknown)
	TPMResetAt time.Time // when the TPM window resets

	// Header-derived remaining counts (from x-ratelimit-remaining)
	RPMRemaining int // -1 if unknown
	TPMRemaining int // -1 if unknown

	// Last updated timestamp
	LastUpdated time.Time

	// Total requests and tokens tracked (lifetime)
	TotalRequests int64
	TotalTokens   int64

	// Number of times the provider was delayed due to near-limit
	ThrottleCount int64
}

RateLimitState holds the current rate limit tracking state for a provider.

type RateLimitedProvider 

type RateLimitedProvider struct {
	// contains filtered or unexported fields
}

RateLimitedProvider wraps a Provider with rate limiting.

func (*RateLimitedProvider) Chat 

func (r *RateLimitedProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

func (*RateLimitedProvider) Name 

func (r *RateLimitedProvider) Name() string

func (*RateLimitedProvider) Ping 

func (r *RateLimitedProvider) Ping(ctx context.Context) error

func (*RateLimitedProvider) StreamChat 

func (r *RateLimitedProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

type RateLimiter 

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter implements a token bucket rate limiter per provider. It limits the number of requests per second to avoid hitting provider rate limits.

func NewRateLimiter 

func NewRateLimiter(defaults RateLimitConfig) *RateLimiter

NewRateLimiter creates a rate limiter with default config applied to all providers.

func (*RateLimiter) SetProviderLimit 

func (rl *RateLimiter) SetProviderLimit(provider string, cfg RateLimitConfig)

SetProviderLimit sets a custom rate limit for a specific provider.

func (*RateLimiter) Wait 

func (rl *RateLimiter) Wait(ctx context.Context, provider string) error

Wait blocks until a request token is available for the given provider. Returns immediately if no rate limit is configured.

type RecordedRequest 

type RecordedRequest struct {
	Messages []GraycodeRouterMessage `json:"messages"`
	Model    string                  `json:"model"`
	System   string                  `json:"system,omitempty"`
	Hash     string                  `json:"hash"`
}

RecordedRequest captures the essential fields of a chat request.

type RecordedResponse 

type RecordedResponse struct {
	Content      string               `json:"content,omitempty"`
	ToolCalls    []ToolCall           `json:"tool_calls,omitempty"`
	Usage        *GraycodeRouterUsage `json:"usage,omitempty"`
	FinishReason string               `json:"finish_reason,omitempty"`
	Error        string               `json:"error,omitempty"`
}

RecordedResponse captures the response from a provider.

type RecorderMode 

type RecorderMode string

RecorderMode controls whether the recorder records new interactions or replays existing ones. 

const (
	// RecordModeRecord always records new interactions from the inner provider.
	RecordModeRecord RecorderMode = "record"
	// RecordModeReplay always replays from the cassette; never calls the inner provider.
	RecordModeReplay RecorderMode = "replay"
	// RecordModeAuto replays if the cassette file exists, otherwise records.
	RecordModeAuto RecorderMode = "auto"
)

type RecorderProvider 

type RecorderProvider struct {
	// contains filtered or unexported fields
}

RecorderProvider wraps any Provider to record or replay LLM interactions. It implements the Provider interface and stores interactions in a Cassette.

func NewRecorderProvider 

func NewRecorderProvider(inner Provider, cassettePath string, mode RecorderMode) (*RecorderProvider, error)

NewRecorderProvider creates a RecorderProvider wrapping inner. In auto mode, if the cassette file exists it loads and replays; otherwise it records. In record mode, a fresh cassette is created. In replay mode, the cassette must exist or an error is returned.

func (*RecorderProvider) Chat 

func (r *RecorderProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat either records a new interaction or replays a stored one.

func (*RecorderProvider) Name 

func (r *RecorderProvider) Name() string

Name returns the inner provider name suffixed with "/recorder".

func (*RecorderProvider) Ping 

func (r *RecorderProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*RecorderProvider) Save 

func (r *RecorderProvider) Save() error

Save writes the cassette to its file path.

func (*RecorderProvider) SetRedactor 

func (r *RecorderProvider) SetRedactor(fn func(string) string)

SetRedactor sets a function that redacts sensitive content before recording.

func (*RecorderProvider) StreamChat 

func (r *RecorderProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat either records a new streaming interaction or replays a stored one. In record mode, the real stream is drained and the accumulated response is saved, then a synthetic stream is created to return to the caller. In replay mode, a synthetic stream is created from the stored response.

type Relationship 

type Relationship struct {
	// Subject is the entity the relation originates from. A noun/noun phrase.
	Subject string `json:"subject"`
	// Predicate is the typed relation (e.g. "depends_on", "authored_by").
	Predicate string `json:"predicate"`
	// Object is the entity the relation points to. A noun/noun phrase.
	Object string `json:"object"`
}

Relationship is a subject-predicate-object triple extracted from text. It is the unit of a knowledge graph: subject and object are entities (nouns), and predicate is the typed relation between them.

This mirrors the typed-extraction pattern popularized by CocoIndex's ExtractByLlm(output_type=list[Relationship]) — instead of free-form text, the model is constrained to emit a list of these triples, validated against a JSON schema with automatic retry.

type RepeatDetector 

type RepeatDetector = core.RepeatDetector

RepeatDetector detects degenerate repeating output in streamed text.

func DefaultRepeatDetector 

func DefaultRepeatDetector() *RepeatDetector

DefaultRepeatDetector returns a RepeatDetector with production thresholds.

type RequestLogEntry 

type RequestLogEntry struct {
	Timestamp    time.Time `json:"timestamp"`
	Provider     string    `json:"provider"`
	Model        string    `json:"model"`
	InputTokens  int       `json:"input_tokens"`
	OutputTokens int       `json:"output_tokens"`
	LatencyMs    int64     `json:"latency_ms"`
	Status       string    `json:"status"` // "success" or "error"
	Error        string    `json:"error,omitempty"`
	CacheHit     bool      `json:"cache_hit"`
}

RequestLogEntry is a single logged API call.

type RequestLogger 

type RequestLogger struct {
	// contains filtered or unexported fields
}

RequestLogger logs all API requests/responses for debugging.

func NewRequestLogger 

func NewRequestLogger(enabled bool) *RequestLogger

NewRequestLogger creates a logger.

func (*RequestLogger) Log 

func (rl *RequestLogger) Log(entry RequestLogEntry)

Log records a request.

func (*RequestLogger) Recent 

func (rl *RequestLogger) Recent(n int) []RequestLogEntry

Recent returns the last N log entries.

func (*RequestLogger) Summary 

func (rl *RequestLogger) Summary() string

Summary returns aggregate stats from the log.

type ResponseFormat 

type ResponseFormat = core.ResponseFormat

ResponseFormat specifies the desired output format for the model response.

type ResponseHealth 

type ResponseHealth = core.ResponseHealth

ResponseHealth classifies whether a provider response carried usable output.

func DetectResponseHealth 

func DetectResponseHealth(sig ResponseSignals) ResponseHealth

DetectResponseHealth classifies a response from stream/response signals.

type ResponseSignals 

type ResponseSignals = core.ResponseSignals

ResponseSignals are the observations needed to classify response health.

type RetryConfig 

type RetryConfig = core.RetryConfig

RetryConfig controls retry behavior for HTTP clients.

func DefaultRetryConfig 

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns sensible defaults.

func NewRetryConfig 

func NewRetryConfig(maxRetries int, baseDelay, maxDelay time.Duration, retryOn ...int) RetryConfig

NewRetryConfig constructs a RetryConfig from core fields and optional HTTP status codes to retry on.

type RoleRouter 

type RoleRouter struct {
	// contains filtered or unexported fields
}

RoleRouter wraps a Provider and overrides ChatOptions.Model with the model configured for the request's role before delegating. The role is taken from the context (see WithRole); when absent, RolePrimary is used. The router only sets a model when the resolved slot is non-empty, so it never clears an explicit opts.Model with an unconfigured role.

RoleRouter follows the same decorator pattern as BudgetProvider and TracingProvider: it is additive and does not change ChatOptions semantics.

func NewRoleRouter 

func NewRoleRouter(inner Provider, roles ModelRoles) (*RoleRouter, error)

NewRoleRouter wraps inner so that requests are routed to the model configured for their role. The inner provider must not be nil; an error is returned otherwise.

func (*RoleRouter) Chat 

func (r *RoleRouter) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat resolves the request's role to a model, applies it to opts, then delegates to the inner provider.

func (*RoleRouter) Name 

func (r *RoleRouter) Name() string

Name returns the inner provider's name.

func (*RoleRouter) Ping 

func (r *RoleRouter) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*RoleRouter) StreamChat 

func (r *RoleRouter) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat resolves the request's role to a model, applies it to opts, then delegates to the inner provider.

type SSEEvent 

type SSEEvent = core.SSEEvent

SSEEvent is one server-sent event from a streaming response body.

type SchemaValidation 

type SchemaValidation struct {
	// Schema is the JSON schema to validate against.
	Schema map[string]interface{}
	// MaxRetries is the maximum number of retry attempts if validation fails.
	MaxRetries int
	// StrictMode enables strict schema validation requiring all fields.
	StrictMode bool
}

SchemaValidation holds configuration for structured output validation with retry.

type SemanticCacheConfig 

type SemanticCacheConfig = embeddings.SemanticCacheConfig

SemanticCacheConfig configures the embedding-based semantic cache.

func DefaultSemanticCacheConfig 

func DefaultSemanticCacheConfig() SemanticCacheConfig

DefaultSemanticCacheConfig returns sensible semantic cache defaults.

type SemanticCacheStats 

type SemanticCacheStats = embeddings.SemanticCacheStats

SemanticCacheStats reports semantic cache effectiveness.

type StepFunClient 

type StepFunClient = adapters.StepFunClient

StepFunClient implements Provider for the StepFun API.

func NewStepFunClient 

func NewStepFunClient(apiKey, openAIBase string, compat *OpenAICompatConfig, opts ...ClientOption) *StepFunClient

type StreamGuardrailConfig 

type StreamGuardrailConfig = core.StreamGuardrailConfig

StreamGuardrailConfig configures incremental guardrail scanning.

type StreamGuardrailResult 

type StreamGuardrailResult = core.StreamGuardrailResult

StreamGuardrailResult is the outcome of scanning one stream chunk.

type StreamGuardrails 

type StreamGuardrails = core.StreamGuardrails

StreamGuardrails applies guardrail rules to a response stream chunk by chunk.

func NewStreamGuardrails 

func NewStreamGuardrails(g *Guardrails, config StreamGuardrailConfig) *StreamGuardrails

NewStreamGuardrails builds an incremental guardrail scanner over a rule set.

type StreamMerger 

type StreamMerger struct {
	StreamFields []string
	IndexFields  []string
	// contains filtered or unexported fields
}

StreamMerger is a schema-agnostic SSE delta merger. It accumulates streaming deltas into a single result map without knowing the provider's schema upfront.

StreamFields are merged by string concatenation (e.g. "content", "arguments"). IndexFields name the key within array elements that identifies their position (e.g. "index" in choices[N].delta), enabling correct out-of-order multi-choice reassembly: element A at index 2 is placed at slot 2 regardless of arrival order.

Pattern ported from moonpalace/merge/merger.go (MIT).

func DefaultStreamMerger 

func DefaultStreamMerger() *StreamMerger

DefaultStreamMerger returns a StreamMerger with Kimi/OpenAI defaults: StreamFields = ["content", "arguments"], IndexFields = ["index"].

func NewStreamMerger 

func NewStreamMerger(streamFields, indexFields []string) *StreamMerger

NewStreamMerger returns a StreamMerger with the given stream and index field names.

func (*StreamMerger) Merge 

func (m *StreamMerger) Merge(delta map[string]interface{}) map[string]interface{}

Merge incorporates a parsed delta map into the accumulated result. It returns the updated accumulator (same map, mutated in place).

func (*StreamMerger) Result 

func (m *StreamMerger) Result() map[string]interface{}

Result returns the current accumulated state.

type StreamResult 

type StreamResult = core.StreamResult

StreamResult wraps a streaming response with cleanup.

func NewStreamResult 

func NewStreamResult(events <-chan GraycodeRouterStreamEvent, cancel context.CancelFunc) *StreamResult

NewStreamResult creates a StreamResult with a cancel function for resource cleanup. The request ID is optional; pass "" when it is not yet available. The canonical constructor lives in github.com/GrayCodeAI/graycode-router/llm; this is a thin facade wrapper that keeps the public client API stable.

func NewStreamResultWithRequestID 

func NewStreamResultWithRequestID(events <-chan GraycodeRouterStreamEvent, requestID string, cancel context.CancelFunc) *StreamResult

NewStreamResultWithRequestID is NewStreamResult carrying the provider's request ID.

func StreamChatWithContinuation 

func StreamChatWithContinuation(ctx context.Context, p Provider, messages []GraycodeRouterMessage, opts ChatOptions, cfg ContinuationConfig) (*StreamResult, error)

StreamChatWithContinuation wraps StreamChat with automatic continuation when the response stops with "max_tokens" and contains only text (no tool calls). It returns a StreamResult whose Events channel transparently continues across multiple LLM calls, emitting a "continuation" event at each boundary.

DEPRECATION NOTE: hawk's Session loop has its own max_tokens recovery (internal/engine/stream.go around the `recoveryCount` loop) that doesn't add a synthetic "Continue." user message, and the graycode-router conversation engine (graycode-router/conversation.Engine) has its own OutputGroupID-based engine-level continuation. The two engine-level paths produce cleaner conversation shapes (no synthetic user turns) and are the recommended pattern for new code. This client-level helper remains for backwards-compatibility with the embedded graycode-router HTTP server and non-hawk consumers; new code should implement continuation at the engine or call-site level instead.

Will be removed in graycode-router v0.3.0. See graycode-router/CHANGELOG.md for the deprecation timeline.

type StreamingTokenCounter 

type StreamingTokenCounter struct {
	// contains filtered or unexported fields
}

StreamingTokenCounter counts tokens as they stream in real-time. Provides running cost estimate during generation.

func NewStreamingTokenCounter 

func NewStreamingTokenCounter(model string, inputTokens int) *StreamingTokenCounter

NewStreamingTokenCounter creates a counter for a specific model.

func (*StreamingTokenCounter) AddCached 

func (stc *StreamingTokenCounter) AddCached(tokens int)

AddCached records cached input tokens.

func (*StreamingTokenCounter) AddOutput 

func (stc *StreamingTokenCounter) AddOutput(text string)

AddOutput records streamed output tokens.

func (*StreamingTokenCounter) CurrentCost 

func (stc *StreamingTokenCounter) CurrentCost() float64

CurrentCost returns the running cost so far.

func (*StreamingTokenCounter) Summary 

func (stc *StreamingTokenCounter) Summary() string

Summary returns current token counts and cost.

type StructuredOutputError 

type StructuredOutputError struct {
	// Response is the raw response that failed validation.
	Response string
	// ValidationErr is the underlying validation error.
	ValidationErr error
	// Attempt is the attempt number that failed.
	Attempt int
}

StructuredOutputError represents a validation failure with details.

func (*StructuredOutputError) Error 

func (e *StructuredOutputError) Error() string

type TokenCountResult 

type TokenCountResult = adapters.TokenCountResult

TokenCountResult holds token counting results.

type ToolCall 

type ToolCall = core.ToolCall

ToolCall represents a tool invocation.

func ParseInlineToolCalls 

func ParseInlineToolCalls(text string) (string, []ToolCall)

ParseInlineToolCalls extracts inline/Hermes-style tool calls from text.

type ToolChoiceOption 

type ToolChoiceOption = core.ToolChoiceOption

ToolChoiceOption controls how the model uses tools (Anthropic).

type ToolResult 

type ToolResult = core.ToolResult

ToolResult represents the result of a tool execution.

type TracingProvider 

type TracingProvider struct {
	// contains filtered or unexported fields
}

TracingProvider wraps a Provider with OpenTelemetry spans for Chat and StreamChat calls. Use NewTracingProvider to create one.

func NewTracingProvider 

func NewTracingProvider(inner Provider) *TracingProvider

NewTracingProvider wraps the given provider with OTel tracing.

func (*TracingProvider) Chat 

func (tp *TracingProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

func (*TracingProvider) Name 

func (tp *TracingProvider) Name() string

func (*TracingProvider) Ping 

func (tp *TracingProvider) Ping(ctx context.Context) error

func (*TracingProvider) StreamChat 

func (tp *TracingProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

type Transcript 

type Transcript struct {
	Text string `json:"text"`
}

Transcript is the response.

type TranscriptionRequest 

type TranscriptionRequest struct {
	Model    string
	File     []byte
	FileName string
	Language string // optional ISO-639-1
	Prompt   string // optional context/hint
}

TranscriptionRequest is the multipart body for /v1/audio/transcriptions. audio is the raw bytes; model is the transcription model.

type UsageEntry 

type UsageEntry struct {
	Tokens    int
	CostUSD   float64
	Timestamp time.Time
	Provider  string
	Model     string
}

UsageEntry represents a single recorded usage event.

type UsageLimitProvider 

type UsageLimitProvider struct {
	// contains filtered or unexported fields
}

UsageLimitProvider wraps any Provider and enforces token/cost budgets via a UsageTracker. It calls CanProceed() before each Chat/StreamChat request and Record() after successful responses.

If the budget is exhausted, calls return a non-nil error immediately without contacting the upstream provider.

UsageLimitProvider is safe for concurrent use (the underlying UsageTracker is internally synchronised).

func NewUsageLimitProvider 

func NewUsageLimitProvider(inner Provider, tracker *UsageTracker) (*UsageLimitProvider, error)

NewUsageLimitProvider wraps inner with budget enforcement via tracker. Both arguments must be non-nil; an error is returned otherwise.

func (*UsageLimitProvider) Chat 

func (u *UsageLimitProvider) Chat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*GraycodeRouterResponse, error)

Chat sends a non-streaming chat request. The call is gated by the usage tracker's CanProceed() and the response tokens are recorded on success.

func (*UsageLimitProvider) Name 

func (u *UsageLimitProvider) Name() string

Name returns the inner provider name suffixed with "/usage-limit".

func (*UsageLimitProvider) Ping 

func (u *UsageLimitProvider) Ping(ctx context.Context) error

Ping delegates directly to the inner provider (budget is not checked).

func (*UsageLimitProvider) StreamChat 

func (u *UsageLimitProvider) StreamChat(ctx context.Context, messages []GraycodeRouterMessage, opts ChatOptions) (*StreamResult, error)

StreamChat sends a streaming chat request. The budget check happens before the stream starts. Usage is recorded once the stream delivers a "usage" event (typically the final chunk).

func (*UsageLimitProvider) Tracker 

func (u *UsageLimitProvider) Tracker() *UsageTracker

Tracker returns the underlying UsageTracker for inspection or configuration.

type UsageSummary 

type UsageSummary struct {
	HourlyTokens     int
	HourlyRemaining  int
	DailyTokens      int
	DailyRemaining   int
	SessionTokens    int
	SessionRemaining int
	DailyCostUSD     float64
	CostRemaining    float64
	HourlyPct        float64
	DailyPct         float64
}

UsageSummary provides a snapshot of current usage across all windows.

type UsageTracker 

type UsageTracker struct {
	DailyLimit   int
	HourlyLimit  int
	SessionLimit int
	CostLimitUSD float64

	Alerts []Alert
	// contains filtered or unexported fields
}

UsageTracker tracks API usage across sessions and prevents surprise bills.

func NewUsageTracker 

func NewUsageTracker() *UsageTracker

NewUsageTracker creates a UsageTracker with sensible defaults.

func (*UsageTracker) CanProceed 

func (u *UsageTracker) CanProceed() (bool, string)

func (*UsageTracker) CheckThresholds 

func (u *UsageTracker) CheckThresholds()

func (*UsageTracker) EstimateRemaining 

func (u *UsageTracker) EstimateRemaining(tokensPerRequest int) int

func (*UsageTracker) FormatSummary 

func (u *UsageTracker) FormatSummary() string

func (*UsageTracker) GetUsage 

func (u *UsageTracker) GetUsage() UsageSummary

func (*UsageTracker) PruneOld 

func (u *UsageTracker) PruneOld()

func (*UsageTracker) Record 

func (u *UsageTracker) Record(tokens int, costUSD float64, provider, model string)

func (*UsageTracker) Reset 

func (u *UsageTracker) Reset()

type VertexClient 

type VertexClient = adapters.VertexClient

VertexClient implements Provider for the Google Vertex AI API.

func NewVertexClient 

func NewVertexClient(projectID, region, token string) *VertexClient

type ZAIClient 

type ZAIClient = adapters.ZAIClient

ZAIClient implements Provider for the Z.AI API.

func NewZAIClient 

func NewZAIClient(apiKey, openAIBase, anthropicBase string, compat *OpenAICompatConfig, providerID string, opts ...ClientOption) *ZAIClient

Source Files

View all Source files

Directories

Path Synopsis
Package core holds the provider contract and the data types shared by every layer of the graycode-router client: adapters, middleware, caching, embeddings, and the client facade itself.
 Package core holds the provider contract and the data types shared by every layer of the graycode-router client: adapters, middleware, caching, embeddings, and the client facade itself.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.