providers

package
v0.0.0-...-5fe9b4e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2026 License: MIT Imports: 30 Imported by: 0

Documentation

Overview

Package providers provides a factory for creating provider instances.

Package providers provides model registry and routing for LLM providers.

Package providers provides a router for multiple LLM providers.

Index

Constants

This section is empty.

Variables

View Source 
var ErrRegistryNotInitialized = fmt.Errorf("model registry has no models: ensure Initialize() or LoadFromCache() is called before using the router")

ErrRegistryNotInitialized is returned when the router is used before the registry has any models.

Functions

func BuildResponsesOutputItems 

func BuildResponsesOutputItems(msg core.ResponseMessage) []core.ResponsesOutputItem

BuildResponsesOutputItems converts a response message into Responses API output items.

func CloneHTTPHeaders 

func CloneHTTPHeaders(src http.Header) map[string][]string

CloneHTTPHeaders returns a detached copy of an http.Header map.

func ConvertChatResponseToResponses 

func ConvertChatResponseToResponses(resp *core.ChatResponse) *core.ResponsesResponse

ConvertChatResponseToResponses converts a ChatResponse to a ResponsesResponse.

func ConvertResponsesContentToChatContent 

func ConvertResponsesContentToChatContent(content any) (any, bool)

ConvertResponsesContentToChatContent maps Responses input content to Chat content. Text-only arrays are flattened to strings for broader provider compatibility. Any non-text part preserves the array form so multimodal payloads survive routing.

func ConvertResponsesInputToMessages 

func ConvertResponsesInputToMessages(input any) ([]core.Message, error)

ConvertResponsesInputToMessages converts a Responses API input payload into Chat API messages.

func ConvertResponsesRequestToChat 

func ConvertResponsesRequestToChat(req *core.ResponsesRequest) (*core.ChatRequest, error)

ConvertResponsesRequestToChat converts a ResponsesRequest to a ChatRequest. It also validates the supported Responses input shapes and returns an error when the request cannot be converted safely.

func CreateOpenAICompatibleFile 

func CreateOpenAICompatibleFile(ctx context.Context, client *llmclient.Client, req *core.FileCreateRequest) (*core.FileObject, error)

CreateOpenAICompatibleFile uploads a file using the OpenAI-compatible multipart files API.

func CreateOpenAICompatibleFileWithPreparer 

func CreateOpenAICompatibleFileWithPreparer(ctx context.Context, client *llmclient.Client, req *core.FileCreateRequest, prepare openAICompatibleRequestPreparer) (*core.FileObject, error)

func DeleteOpenAICompatibleFile 

func DeleteOpenAICompatibleFile(ctx context.Context, client *llmclient.Client, id string) (*core.FileDeleteResponse, error)

DeleteOpenAICompatibleFile deletes a file object by id after normalizing the incoming id via validatedOpenAICompatibleFileID. Missing response ID and Object fields are synthesized on the returned delete response.

func DeleteOpenAICompatibleFileWithPreparer 

func DeleteOpenAICompatibleFileWithPreparer(ctx context.Context, client *llmclient.Client, id string, prepare openAICompatibleRequestPreparer) (*core.FileDeleteResponse, error)

func EnsureResponsesDone 

func EnsureResponsesDone(stream io.ReadCloser) io.ReadCloser

EnsureResponsesDone normalizes Responses API streams so clients always receive a terminal data: [DONE] marker when the upstream stream reaches a completed Responses event but closes at EOF before sending the final marker.

func ExtractContentFromInput 

func ExtractContentFromInput(content any) string

ExtractContentFromInput extracts text content from responses input.

func FetchBatchResultsFromOutputFile 

func FetchBatchResultsFromOutputFile(ctx context.Context, client *llmclient.Client, providerName, batchID string) (*core.BatchResultsResponse, error)

FetchBatchResultsFromOutputFile adapts OpenAI-compatible batch output files to gateway batch results.

func FetchBatchResultsFromOutputFileWithPreparer 

func FetchBatchResultsFromOutputFileWithPreparer(ctx context.Context, client *llmclient.Client, providerName, batchID string, prepare openAICompatibleRequestPreparer) (*core.BatchResultsResponse, error)

func GetOpenAICompatibleFile 

func GetOpenAICompatibleFile(ctx context.Context, client *llmclient.Client, id string) (*core.FileObject, error)

GetOpenAICompatibleFile retrieves a file object by id after normalizing the incoming id via validatedOpenAICompatibleFileID. Missing response ID and Object fields are synthesized on the returned file object.

func GetOpenAICompatibleFileContent 

func GetOpenAICompatibleFileContent(ctx context.Context, client *llmclient.Client, id string) (*core.FileContentResponse, error)

GetOpenAICompatibleFileContent fetches file bytes via /files/{id}/content after normalizing the incoming id via validatedOpenAICompatibleFileID. The returned response always includes the normalized file ID.

func GetOpenAICompatibleFileContentWithPreparer 

func GetOpenAICompatibleFileContentWithPreparer(ctx context.Context, client *llmclient.Client, id string, prepare openAICompatibleRequestPreparer) (*core.FileContentResponse, error)

func GetOpenAICompatibleFileWithPreparer 

func GetOpenAICompatibleFileWithPreparer(ctx context.Context, client *llmclient.Client, id string, prepare openAICompatibleRequestPreparer) (*core.FileObject, error)

func ListOpenAICompatibleFiles 

func ListOpenAICompatibleFiles(ctx context.Context, client *llmclient.Client, purpose string, limit int, after string) (*core.FileListResponse, error)

ListOpenAICompatibleFiles lists files using OpenAI-compatible files API.

func ListOpenAICompatibleFilesWithPreparer 

func ListOpenAICompatibleFilesWithPreparer(ctx context.Context, client *llmclient.Client, purpose string, limit int, after string, prepare openAICompatibleRequestPreparer) (*core.FileListResponse, error)

func PassthroughEndpoint 

func PassthroughEndpoint(endpoint string) string

PassthroughEndpoint normalizes a provider-relative passthrough endpoint into an absolute path fragment suitable for baseURL + endpoint request building.

func PassthroughEndpointPath 

func PassthroughEndpointPath(info *core.PassthroughRouteInfo) string

PassthroughEndpointPath returns the normalized path portion of a provider passthrough endpoint, preferring a semantic normalized endpoint when present.

func ResolveAPIVersion 

func ResolveAPIVersion(apiVersion, fallback string) string

ResolveAPIVersion returns the configured API version when present, otherwise the provider default.

func ResolveBaseURL 

func ResolveBaseURL(baseURL, fallback string) string

ResolveBaseURL returns the configured base URL when present, otherwise the provider default.

func ResponsesFunctionCallCallID 

func ResponsesFunctionCallCallID(callID string) string

ResponsesFunctionCallCallID returns the call id if present or generates one.

func ResponsesFunctionCallItemID 

func ResponsesFunctionCallItemID(callID string) string

ResponsesFunctionCallItemID returns a stable function-call item id.

func ResponsesViaChat 

func ResponsesViaChat(ctx context.Context, p ChatProvider, req *core.ResponsesRequest) (*core.ResponsesResponse, error)

ResponsesViaChat implements the Responses API by converting to/from Chat format.

func StreamResponsesViaChat 

func StreamResponsesViaChat(ctx context.Context, p ChatProvider, req *core.ResponsesRequest, providerName string) (io.ReadCloser, error)

StreamResponsesViaChat implements streaming Responses API by converting to/from Chat format.

Types

type CategoryCount 

type CategoryCount struct {
	Category    core.ModelCategory `json:"category"`
	DisplayName string             `json:"display_name"`
	Count       int                `json:"count"`
}

CategoryCount holds a model category and the number of models in it.

type ChatProvider 

type ChatProvider interface {
	ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)
	StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)
}

ChatProvider is the minimal interface needed by the shared Responses-to-Chat adapter. Any provider that supports ChatCompletion and StreamChatCompletion can use the ResponsesViaChat and StreamResponsesViaChat helpers to implement the Responses API.

type DiscoveryConfig 

type DiscoveryConfig struct {
	DefaultBaseURL     string
	RequireBaseURL     bool
	AllowAPIKeyless    bool
	SupportsAPIVersion bool
}

DiscoveryConfig describes how a provider participates in config resolution. Env var names are derived by convention from Registration.Type.

type InitResult 

type InitResult struct {
	Registry *ModelRegistry
	Router   *Router
	Cache    modelcache.Cache
	Factory  *ProviderFactory

	// ConfiguredProviders is the effective, admin-safe provider inventory keyed
	// by configured provider name.
	ConfiguredProviders []SanitizedProviderConfig

	// CredentialResolvedProviders is the env-merged, credential-filtered providers
	// map (same keys as Router). Keys match top-level providers YAML names.
	CredentialResolvedProviders map[string]config.RawProviderConfig
	// contains filtered or unexported fields
}

InitResult holds the initialized provider infrastructure and cleanup functions.

func Init 

func Init(ctx context.Context, result *config.LoadResult, factory *ProviderFactory) (*InitResult, error)

Init initializes the provider registry, cache, and router.

It performs:

  1. Provider config resolution (env var overlay, filtering, resilience merging)
  2. Cache initialization (local or Redis based on config)
  3. Provider instantiation and registration
  4. Async model loading (from cache first, then network refresh)
  5. Best-effort background model-list fetch (goroutine with ~45s timeout that calls modeldata.Fetch, registry.EnrichModels, and SaveToCache)
  6. Background refresh scheduling (interval from cfg.Cache.RefreshInterval)
  7. Router creation

The caller must call InitResult.Close() during shutdown.

func (*InitResult) Close 

func (r *InitResult) Close() error

Close releases all resources and stops background goroutines. Safe to call multiple times (but stopRefresh is only called once).

type ModelInfo 

type ModelInfo struct {
	Model        core.Model
	Provider     core.Provider
	ProviderName string
	ProviderType string
}

ModelInfo holds information about a model and its provider

type ModelRegistry 

type ModelRegistry struct {
	// contains filtered or unexported fields
}

ModelRegistry manages the mapping of models to their providers. It fetches models from providers on startup and caches them in memory. Supports loading from a cache (local file or Redis) for instant startup.

func NewModelRegistry 

func NewModelRegistry() *ModelRegistry

NewModelRegistry creates a new model registry

func (*ModelRegistry) EnrichModels 

func (r *ModelRegistry) EnrichModels()

EnrichModels re-applies model list metadata to all currently registered models. Call this after SetModelList to update existing models with the new metadata. Holds the write lock for the entire operation and replaces published ModelInfo entries instead of mutating them in place so concurrent readers can safely keep using older snapshots after unlocking.

func (*ModelRegistry) GetCategoryCounts 

func (r *ModelRegistry) GetCategoryCounts() []CategoryCount

GetCategoryCounts returns model counts per category, in display order. A model with multiple categories is counted in each.

func (*ModelRegistry) GetModel 

func (r *ModelRegistry) GetModel(model string) *ModelInfo

GetModel returns the registry-backed model info for the given model, or nil if not found. Callers must treat the returned data as read-only.

func (*ModelRegistry) GetModelMetadata 

func (r *ModelRegistry) GetModelMetadata(modelID string) *core.ModelMetadata

GetModelMetadata returns the metadata for a model, or nil if not found or not enriched.

func (*ModelRegistry) GetProvider 

func (r *ModelRegistry) GetProvider(model string) core.Provider

GetProvider returns the provider for the given model, or nil if not found

func (*ModelRegistry) GetProviderName 

func (r *ModelRegistry) GetProviderName(model string) string

GetProviderName returns the concrete configured provider instance name for the given model selector. Returns empty string if the model is not found.

func (*ModelRegistry) GetProviderNameForType 

func (r *ModelRegistry) GetProviderNameForType(providerType string) string

GetProviderNameForType returns the first registered configured provider name for the given provider type. This follows the same first-registered behavior used when provider-typed routes resolve a concrete provider instance.

func (*ModelRegistry) GetProviderType 

func (r *ModelRegistry) GetProviderType(model string) string

GetProviderType returns the provider type string for the given model. Returns empty string if the model is not found.

func (*ModelRegistry) GetProviderTypeForName 

func (r *ModelRegistry) GetProviderTypeForName(providerName string) string

GetProviderTypeForName returns the provider type for the given concrete configured provider instance name.

func (*ModelRegistry) Initialize 

func (r *ModelRegistry) Initialize(ctx context.Context) error

Initialize fetches models from all registered providers and populates the registry. This should be called on application startup.

func (*ModelRegistry) InitializeAsync 

func (r *ModelRegistry) InitializeAsync(ctx context.Context)

InitializeAsync starts model fetching in a background goroutine. It first loads any cached models for immediate availability, then refreshes from network. Returns immediately after loading cache. The background goroutine will update models and save to cache when network fetch completes.

func (*ModelRegistry) IsInitialized 

func (r *ModelRegistry) IsInitialized() bool

IsInitialized returns true if at least one successful network fetch has completed. This can be used to check if the registry has fresh data or is only serving from cache.

func (*ModelRegistry) ListModels 

func (r *ModelRegistry) ListModels() []core.Model

ListModels returns all models in the registry, sorted by model ID for consistent ordering. The sorted slice is cached and rebuilt only when the underlying models change. Returns a defensive copy so callers cannot mutate the internal cache.

func (*ModelRegistry) ListModelsWithProvider 

func (r *ModelRegistry) ListModelsWithProvider() []ModelWithProvider

ListModelsWithProvider returns all provider-backed models with provider metadata, sorted by public selector. The sorted slice is cached and rebuilt only when the underlying models change. Returns a defensive copy so callers cannot mutate the internal cache.

func (*ModelRegistry) ListModelsWithProviderByCategory 

func (r *ModelRegistry) ListModelsWithProviderByCategory(category core.ModelCategory) []ModelWithProvider

ListModelsWithProviderByCategory returns provider-backed models filtered by category, sorted by public selector. If category is CategoryAll, returns all models (same as ListModelsWithProvider). Results for known categories are cached and rebuilt only when the underlying models change. Returns a defensive copy so callers cannot mutate the internal cache.

func (*ModelRegistry) ListPublicModels 

func (r *ModelRegistry) ListPublicModels() []core.Model

ListPublicModels returns all provider-backed models as public selectors in providerName/modelID form, sorted by public model ID.

func (*ModelRegistry) LoadFromCache 

func (r *ModelRegistry) LoadFromCache(ctx context.Context) (int, error)

LoadFromCache loads the model list from the cache backend. Returns the number of models loaded and any error encountered.

func (*ModelRegistry) LookupModel 

func (r *ModelRegistry) LookupModel(model string) (*core.Model, bool)

LookupModel returns a shallow copy of the concrete model for the given selector. Qualified selectors use the configured provider name prefix when present.

func (*ModelRegistry) ModelCount 

func (r *ModelRegistry) ModelCount() int

ModelCount returns the number of registered models

func (*ModelRegistry) ProviderByName 

func (r *ModelRegistry) ProviderByName(providerName string) core.Provider

ProviderByName returns the registered provider for a configured provider instance name.

func (*ModelRegistry) ProviderByType 

func (r *ModelRegistry) ProviderByType(providerType string) core.Provider

ProviderByType returns the first registered provider for the given provider type. This lookup is independent of discovered models so provider-typed routes keep working even when a provider currently exposes zero models.

func (*ModelRegistry) ProviderCount 

func (r *ModelRegistry) ProviderCount() int

ProviderCount returns the number of registered providers

func (*ModelRegistry) ProviderNames 

func (r *ModelRegistry) ProviderNames() []string

ProviderNames returns the configured provider instance names in registration order.

func (*ModelRegistry) ProviderRuntimeSnapshots 

func (r *ModelRegistry) ProviderRuntimeSnapshots() []ProviderRuntimeSnapshot

ProviderRuntimeSnapshots returns runtime diagnostics for configured providers keyed by configured provider name.

func (*ModelRegistry) ProviderTypes 

func (r *ModelRegistry) ProviderTypes() []string

ProviderTypes returns the unique registered provider types in sorted order. This inventory is independent of discovered models.

func (*ModelRegistry) RecordAvailabilityCheck 

func (r *ModelRegistry) RecordAvailabilityCheck(providerName string, err error)

RecordAvailabilityCheck stores the latest startup or explicit availability probe result for a configured provider name.

func (*ModelRegistry) Refresh 

func (r *ModelRegistry) Refresh(ctx context.Context) error

Refresh updates the model registry by fetching fresh model lists from providers. This can be called periodically to keep the registry up to date.

func (*ModelRegistry) RefreshModelList 

func (r *ModelRegistry) RefreshModelList(ctx context.Context, url string) (int, error)

RefreshModelList fetches the external model metadata list and re-enriches all currently registered models. It does not persist the model cache; callers that want durable startup data should call SaveToCache after this succeeds.

func (*ModelRegistry) RegisterProvider 

func (r *ModelRegistry) RegisterProvider(provider core.Provider)

RegisterProvider adds a provider to the registry

func (*ModelRegistry) RegisterProviderWithNameAndType 

func (r *ModelRegistry) RegisterProviderWithNameAndType(provider core.Provider, providerName, providerType string)

RegisterProviderWithNameAndType adds a provider with a configured provider instance name and type. Name is used for unambiguous provider/model selection (e.g. "provider/model") and cache persistence.

func (*ModelRegistry) RegisterProviderWithType 

func (r *ModelRegistry) RegisterProviderWithType(provider core.Provider, providerType string)

RegisterProviderWithType adds a provider to the registry with its type string. The type is used for cache persistence to re-associate models with providers on startup.

func (*ModelRegistry) ResolveMetadata 

func (r *ModelRegistry) ResolveMetadata(providerType, modelID string) *core.ModelMetadata

ResolveMetadata resolves metadata for a model directly via the stored model list, bypassing the registry key lookup. This handles cases where the usage DB stores a response model ID (e.g., "gpt-4o-2024-08-06") that differs from the registry key (e.g., "gpt-4o") by using the reverse index in the model list.

func (*ModelRegistry) ResolvePricing 

func (r *ModelRegistry) ResolvePricing(model, providerType string) *core.ModelPricing

ResolvePricing returns the pricing metadata for a model, trying the registry first and falling back to a reverse-index lookup via the model list. Returns nil if no pricing is available.

func (*ModelRegistry) SaveToCache 

func (r *ModelRegistry) SaveToCache(ctx context.Context) error

SaveToCache saves the current model list to the cache backend.

func (*ModelRegistry) SetCache 

func (r *ModelRegistry) SetCache(c modelcache.Cache)

SetCache sets the cache backend for persistent model storage. The cache can be a local file-based cache or a Redis cache.

func (*ModelRegistry) SetConfiguredProviderModelsMode 

func (r *ModelRegistry) SetConfiguredProviderModelsMode(mode config.ConfiguredProviderModelsMode)

SetConfiguredProviderModelsMode controls how configured provider model lists affect the final registry inventory.

func (*ModelRegistry) SetModelList 

func (r *ModelRegistry) SetModelList(list *modeldata.ModelList, raw json.RawMessage)

SetModelList stores the parsed model list and its raw bytes for cache persistence.

func (*ModelRegistry) SetProviderConfiguredModels 

func (r *ModelRegistry) SetProviderConfiguredModels(providerName string, models []string)

SetProviderConfiguredModels records the explicit model inventory declared for a configured provider instance. Call with an empty/nil slice to clear it.

func (*ModelRegistry) SetProviderMetadataOverrides 

func (r *ModelRegistry) SetProviderMetadataOverrides(providerName string, overrides map[string]*core.ModelMetadata)

SetProviderMetadataOverrides records per-model metadata overrides declared in config.yaml for the given provider instance name. Overrides are merged onto remote-registry enrichment each time the registry re-enriches its models.

Call with an empty/nil map to clear any prior overrides for that provider.

func (*ModelRegistry) StartBackgroundRefresh 

func (r *ModelRegistry) StartBackgroundRefresh(interval time.Duration, modelListURL string) func()

StartBackgroundRefresh starts a goroutine that periodically refreshes the model registry. If modelListURL is non-empty, the model list is also re-fetched on each tick. The returned stop function is blocking: it cancels the refresh loop and waits for the goroutine to exit before returning, so callers should expect it to block during shutdown until any in-flight refresh work unwinds.

func (*ModelRegistry) Supports 

func (r *ModelRegistry) Supports(model string) bool

Supports returns true if the registry has a provider for the given model

type ModelWithProvider 

type ModelWithProvider struct {
	Model        core.Model `json:"model"`
	ProviderType string     `json:"provider_type"`
	ProviderName string     `json:"provider_name"`
	Selector     string     `json:"selector"`
}

ModelWithProvider holds a model alongside provider metadata and its public selector.

type OpenAIResponsesStreamConverter 

type OpenAIResponsesStreamConverter struct {
	// contains filtered or unexported fields
}

OpenAIResponsesStreamConverter wraps an OpenAI-compatible SSE stream and converts it to Responses API format. Used by providers that have OpenAI-compatible streaming (Groq, Gemini, etc.)

func NewOpenAIResponsesStreamConverter 

func NewOpenAIResponsesStreamConverter(reader io.ReadCloser, model, provider string) *OpenAIResponsesStreamConverter

NewOpenAIResponsesStreamConverter creates a new converter that transforms OpenAI-format SSE streams to Responses API format.

func (*OpenAIResponsesStreamConverter) Close 

func (sc *OpenAIResponsesStreamConverter) Close() error

func (*OpenAIResponsesStreamConverter) Read 

func (sc *OpenAIResponsesStreamConverter) Read(p []byte) (n int, err error)

type ProviderConfig 

type ProviderConfig struct {
	Type       string
	APIKey     string
	BaseURL    string
	APIVersion string
	Models     []string
	// ModelMetadataOverrides holds operator-supplied metadata keyed by raw model
	// ID (as it appears in the provider's /models response). The registry merges
	// these onto remote-registry metadata after enrichment; non-zero fields here
	// win. Empty/nil when no per-model metadata is declared in YAML.
	ModelMetadataOverrides map[string]*core.ModelMetadata
	Resilience             config.ResilienceConfig
}

ProviderConfig holds the fully resolved provider configuration after merging global defaults with per-provider overrides.

type ProviderConstructor 

type ProviderConstructor func(cfg ProviderConfig, opts ProviderOptions) core.Provider

ProviderConstructor is the constructor signature for providers.

type ProviderFactory 

type ProviderFactory struct {
	// contains filtered or unexported fields
}

ProviderFactory manages provider registration and creation.

func NewProviderFactory 

func NewProviderFactory() *ProviderFactory

NewProviderFactory creates a new provider factory instance.

func (*ProviderFactory) Add 

func (f *ProviderFactory) Add(reg Registration)

Add adds a provider constructor to the factory. Panics if reg.Type is empty or reg.New is nil — both are programming errors caught at startup, not runtime conditions.

func (*ProviderFactory) Create 

func (f *ProviderFactory) Create(cfg ProviderConfig) (core.Provider, error)

Create instantiates a provider based on its resolved configuration.

func (*ProviderFactory) PassthroughSemanticEnrichers 

func (f *ProviderFactory) PassthroughSemanticEnrichers() []core.PassthroughSemanticEnricher

PassthroughSemanticEnrichers returns registered passthrough semantic enrichers in deterministic provider-type order.

func (*ProviderFactory) RegisteredTypes 

func (f *ProviderFactory) RegisteredTypes() []string

RegisteredTypes returns a list of all registered provider types.

func (*ProviderFactory) SetHooks 

func (f *ProviderFactory) SetHooks(hooks llmclient.Hooks)

SetHooks configures observability hooks for all providers created by this factory.

type ProviderOptions 

type ProviderOptions struct {
	Hooks      llmclient.Hooks
	Models     []string
	Resilience config.ResilienceConfig
}

ProviderOptions bundles runtime settings passed from the factory to provider constructors.

type ProviderRuntimeSnapshot 

type ProviderRuntimeSnapshot struct {
	Name                    string     `json:"name"`
	Type                    string     `json:"type"`
	Registered              bool       `json:"registered"`
	RegistryInitialized     bool       `json:"registry_initialized"`
	DiscoveredModelCount    int        `json:"discovered_model_count"`
	UsingCachedModels       bool       `json:"using_cached_models"`
	LastModelFetchAt        *time.Time `json:"last_model_fetch_at,omitempty"`
	LastModelFetchSuccessAt *time.Time `json:"last_model_fetch_success_at,omitempty"`
	LastModelFetchError     string     `json:"last_model_fetch_error,omitempty"`
	LastAvailabilityCheckAt *time.Time `json:"last_availability_check_at,omitempty"`
	LastAvailabilityOKAt    *time.Time `json:"last_availability_ok_at,omitempty"`
	LastAvailabilityError   string     `json:"last_availability_error,omitempty"`
}

ProviderRuntimeSnapshot describes runtime diagnostics for a configured provider.

type Registration 

type Registration struct {
	Type                        string
	New                         ProviderConstructor
	PassthroughSemanticEnricher core.PassthroughSemanticEnricher
	Discovery                   DiscoveryConfig
}

Registration contains metadata for registering a provider with the factory.

type ResponsesOutputEventState 

type ResponsesOutputEventState struct {
	// contains filtered or unexported fields
}

ResponsesOutputEventState manages assistant/tool output items for Responses streams.

func NewResponsesOutputEventState 

func NewResponsesOutputEventState(responseID string) *ResponsesOutputEventState

NewResponsesOutputEventState creates a new Responses output-item state manager.

func (*ResponsesOutputEventState) AppendAssistantText 

func (s *ResponsesOutputEventState) AppendAssistantText(text string)

AppendAssistantText appends assistant text content to the output item buffer.

func (*ResponsesOutputEventState) AssistantDone 

func (s *ResponsesOutputEventState) AssistantDone() bool

AssistantDone reports whether the assistant output item has been completed.

func (*ResponsesOutputEventState) AssistantMessageItem 

func (s *ResponsesOutputEventState) AssistantMessageItem(status string, includeContent bool) map[string]any

AssistantMessageItem renders the assistant message output item payload.

func (*ResponsesOutputEventState) AssistantReserved 

func (s *ResponsesOutputEventState) AssistantReserved() bool

AssistantReserved reports whether the assistant output item has been reserved.

func (*ResponsesOutputEventState) AssistantStarted 

func (s *ResponsesOutputEventState) AssistantStarted() bool

AssistantStarted reports whether the assistant output item has been emitted.

func (*ResponsesOutputEventState) CompleteAssistantOutput 

func (s *ResponsesOutputEventState) CompleteAssistantOutput(outputIndex int) string

CompleteAssistantOutput emits the assistant message output_item.done event once.

func (*ResponsesOutputEventState) CompleteToolCall 

func (s *ResponsesOutputEventState) CompleteToolCall(state *ResponsesOutputToolCallState, includePlaceholder bool) string

CompleteToolCall emits the argument completion and output_item.done events once.

func (*ResponsesOutputEventState) RenderToolCallItem 

func (s *ResponsesOutputEventState) RenderToolCallItem(state *ResponsesOutputToolCallState, status string, includePlaceholder bool) map[string]any

RenderToolCallItem renders a function_call output item payload.

func (*ResponsesOutputEventState) ReserveAssistant 

func (s *ResponsesOutputEventState) ReserveAssistant()

ReserveAssistant marks that the assistant message output item occupies index 0.

func (*ResponsesOutputEventState) StartAssistantOutput 

func (s *ResponsesOutputEventState) StartAssistantOutput(outputIndex int) string

StartAssistantOutput emits the assistant message output_item.added event once.

func (*ResponsesOutputEventState) StartToolCall 

func (s *ResponsesOutputEventState) StartToolCall(state *ResponsesOutputToolCallState, includePlaceholder bool) string

StartToolCall emits the function_call output_item.added event once the item metadata is available.

func (*ResponsesOutputEventState) ToolCallArguments 

func (s *ResponsesOutputEventState) ToolCallArguments(state *ResponsesOutputToolCallState) string

ToolCallArguments returns the serialized argument payload for a function_call item.

func (*ResponsesOutputEventState) WriteEvent 

func (s *ResponsesOutputEventState) WriteEvent(eventName string, payload map[string]any) string

WriteEvent renders one SSE event in Responses API format.

type ResponsesOutputToolCallState 

type ResponsesOutputToolCallState struct {
	ItemID            string
	CallID            string
	Name              string
	OutputIndex       int
	Arguments         strings.Builder
	Started           bool
	Completed         bool
	PlaceholderObject bool
}

ResponsesOutputToolCallState tracks one function_call item in a Responses stream.

type Router 

type Router struct {
	// contains filtered or unexported fields
}

Router routes requests to the appropriate provider based on the model lookup. It uses a dynamic model-to-provider mapping that is populated at startup by fetching available models from each provider's /models endpoint.

func NewRouter 

func NewRouter(lookup core.ModelLookup) (*Router, error)

NewRouter creates a new provider router with a model lookup. The lookup must be initialized (via Initialize() or LoadFromCache()) before using the router. Returns an error if the lookup is nil.

func (*Router) CancelBatch 

func (r *Router) CancelBatch(ctx context.Context, providerType, id string) (*core.BatchResponse, error)

CancelBatch routes native batch cancellation to a provider type.

func (*Router) CancelResponse 

func (r *Router) CancelResponse(ctx context.Context, providerType, id string) (*core.ResponsesResponse, error)

CancelResponse routes native response cancellation to a provider type.

func (*Router) ChatCompletion 

func (r *Router) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)

ChatCompletion routes the request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) ClearBatchResultHints 

func (r *Router) ClearBatchResultHints(providerType, batchID string)

ClearBatchResultHints clears transient provider-side batch result hints once they have been persisted by the gateway.

func (*Router) CompactResponse 

func (r *Router) CompactResponse(ctx context.Context, providerType string, req *core.ResponsesRequest) (*core.ResponseCompactResponse, error)

CompactResponse routes native response compaction to a provider type.

func (*Router) CountResponseInputTokens 

func (r *Router) CountResponseInputTokens(ctx context.Context, providerType string, req *core.ResponsesRequest) (*core.ResponseInputTokensResponse, error)

CountResponseInputTokens routes native response input token counting to a provider type.

func (*Router) CreateBatch 

func (r *Router) CreateBatch(ctx context.Context, providerType string, req *core.BatchRequest) (*core.BatchResponse, error)

CreateBatch routes native batch creation to a provider type.

func (*Router) CreateBatchWithHints 

func (r *Router) CreateBatchWithHints(ctx context.Context, providerType string, req *core.BatchRequest) (*core.BatchResponse, map[string]string, error)

CreateBatchWithHints routes native batch creation and returns any provider batch-result shaping hints that need gateway persistence.

func (*Router) CreateFile 

func (r *Router) CreateFile(ctx context.Context, providerType string, req *core.FileCreateRequest) (*core.FileObject, error)

CreateFile routes file upload to a provider type.

func (*Router) DeleteFile 

func (r *Router) DeleteFile(ctx context.Context, providerType, id string) (*core.FileDeleteResponse, error)

DeleteFile routes file deletion to a provider type.

func (*Router) DeleteResponse 

func (r *Router) DeleteResponse(ctx context.Context, providerType, id string) (*core.ResponseDeleteResponse, error)

DeleteResponse routes native response deletion to a provider type.

func (*Router) Embeddings 

func (r *Router) Embeddings(ctx context.Context, req *core.EmbeddingRequest) (*core.EmbeddingResponse, error)

Embeddings routes the embeddings request to the appropriate provider.

func (*Router) GetBatch 

func (r *Router) GetBatch(ctx context.Context, providerType, id string) (*core.BatchResponse, error)

GetBatch routes native batch lookup to a provider type.

func (*Router) GetBatchResults 

func (r *Router) GetBatchResults(ctx context.Context, providerType, id string) (*core.BatchResultsResponse, error)

GetBatchResults routes native batch results lookup to a provider type.

func (*Router) GetBatchResultsWithHints 

func (r *Router) GetBatchResultsWithHints(ctx context.Context, providerType, id string, endpointByCustomID map[string]string) (*core.BatchResultsResponse, error)

GetBatchResultsWithHints routes native batch results lookup with persisted per-item endpoint hints when the provider supports them.

func (*Router) GetFile 

func (r *Router) GetFile(ctx context.Context, providerType, id string) (*core.FileObject, error)

GetFile routes file retrieval to a provider type.

func (*Router) GetFileContent 

func (r *Router) GetFileContent(ctx context.Context, providerType, id string) (*core.FileContentResponse, error)

GetFileContent routes file content retrieval to a provider type.

func (*Router) GetProviderName 

func (r *Router) GetProviderName(model string) string

GetProviderName returns the concrete configured provider instance name for the given model selector. Returns empty string when unavailable.

func (*Router) GetProviderNameForType 

func (r *Router) GetProviderNameForType(providerType string) string

GetProviderNameForType returns the concrete configured provider instance name chosen for a provider-typed route.

func (*Router) GetProviderType 

func (r *Router) GetProviderType(model string) string

GetProviderType returns the provider type string for the given model. Returns empty string if the model is not found.

func (*Router) GetProviderTypeForName 

func (r *Router) GetProviderTypeForName(providerName string) string

GetProviderTypeForName returns the provider type for a concrete configured provider instance name.

func (*Router) GetResponse 

func (r *Router) GetResponse(ctx context.Context, providerType, id string, params core.ResponseRetrieveParams) (*core.ResponsesResponse, error)

GetResponse routes native response retrieval to a provider type.

func (*Router) ListBatches 

func (r *Router) ListBatches(ctx context.Context, providerType string, limit int, after string) (*core.BatchListResponse, error)

ListBatches routes native batch listing to a provider type.

func (*Router) ListFiles 

func (r *Router) ListFiles(ctx context.Context, providerType, purpose string, limit int, after string) (*core.FileListResponse, error)

ListFiles routes file listing to a provider type.

func (*Router) ListModels 

func (r *Router) ListModels(_ context.Context) (*core.ModelsResponse, error)

ListModels returns all models from the lookup. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) ListResponseInputItems 

func (r *Router) ListResponseInputItems(ctx context.Context, providerType, id string, params core.ResponseInputItemsParams) (*core.ResponseInputItemListResponse, error)

ListResponseInputItems routes native response input item listing to a provider type.

func (*Router) ModelCount 

func (r *Router) ModelCount() int

ModelCount returns the number of models currently loaded into the router lookup.

func (*Router) NativeFileProviderTypes 

func (r *Router) NativeFileProviderTypes() []string

NativeFileProviderTypes returns the registered provider types that support native file operations. This inventory is independent of the public model catalog whenever the underlying lookup can expose provider types directly.

func (*Router) NativeResponseProviderTypes 

func (r *Router) NativeResponseProviderTypes() []string

NativeResponseProviderTypes returns the registered provider types that support native Responses lifecycle operations.

func (*Router) Passthrough 

func (r *Router) Passthrough(ctx context.Context, providerType string, req *core.PassthroughRequest) (*core.PassthroughResponse, error)

Passthrough routes an opaque provider-native request by provider type.

func (*Router) ResolveModel 

func (r *Router) ResolveModel(requested core.RequestedModelSelector) (core.ModelSelector, bool, error)

ResolveModel canonicalizes a requested selector into the concrete provider-name-qualified selector used for execution.

Resolution precedence is:

  1. configured provider name + model ID
  2. provider type + model ID
  3. raw slash-shaped model ID (only when provider was not explicit)
  4. default normalization fallback

func (*Router) Responses 

func (r *Router) Responses(ctx context.Context, req *core.ResponsesRequest) (*core.ResponsesResponse, error)

Responses routes the Responses API request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) StreamChatCompletion 

func (r *Router) StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)

StreamChatCompletion routes the streaming request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) StreamResponses 

func (r *Router) StreamResponses(ctx context.Context, req *core.ResponsesRequest) (io.ReadCloser, error)

StreamResponses routes the streaming Responses API request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) Supports 

func (r *Router) Supports(model string) bool

Supports returns true if any provider supports the given model. Returns false if the lookup has no models loaded.

type SanitizedCircuitBreakerConfig 

type SanitizedCircuitBreakerConfig struct {
	FailureThreshold int    `json:"failure_threshold"`
	SuccessThreshold int    `json:"success_threshold"`
	Timeout          string `json:"timeout"`
}

SanitizedCircuitBreakerConfig exposes effective circuit-breaker settings.

type SanitizedProviderConfig 

type SanitizedProviderConfig struct {
	Name       string                    `json:"name"`
	Type       string                    `json:"type"`
	BaseURL    string                    `json:"base_url,omitempty"`
	APIVersion string                    `json:"api_version,omitempty"`
	Models     []string                  `json:"models,omitempty"`
	Resilience SanitizedResilienceConfig `json:"resilience"`
}

SanitizedProviderConfig is the admin-safe provider configuration view.

func SanitizeProviderConfigs 

func SanitizeProviderConfigs(configs map[string]ProviderConfig) []SanitizedProviderConfig

SanitizeProviderConfigs converts effective provider configs into a stable, admin-safe slice keyed by configured provider name.

type SanitizedResilienceConfig 

type SanitizedResilienceConfig struct {
	Retry          SanitizedRetryConfig          `json:"retry"`
	CircuitBreaker SanitizedCircuitBreakerConfig `json:"circuit_breaker"`
}

SanitizedResilienceConfig exposes effective resilience settings.

type SanitizedRetryConfig 

type SanitizedRetryConfig struct {
	MaxRetries     int     `json:"max_retries"`
	InitialBackoff string  `json:"initial_backoff"`
	MaxBackoff     string  `json:"max_backoff"`
	BackoffFactor  float64 `json:"backoff_factor"`
	JitterFactor   float64 `json:"jitter_factor"`
}

SanitizedRetryConfig exposes effective retry settings without secrets.

Source Files

View all Source files

Directories

Path Synopsis
Package anthropic provides Anthropic API integration for the LLM gateway.
 Package anthropic provides Anthropic API integration for the LLM gateway.
Package gemini provides Google Gemini API integration for the LLM gateway.
 Package gemini provides Google Gemini API integration for the LLM gateway.
Package groq provides Groq API integration for the LLM gateway.
 Package groq provides Groq API integration for the LLM gateway.
Package minimax provides MiniMax API integration for the LLM gateway.
 Package minimax provides MiniMax API integration for the LLM gateway.
Package ollama provides Ollama API integration for the LLM gateway.
 Package ollama provides Ollama API integration for the LLM gateway.
Package openai provides OpenAI API integration for the LLM gateway.
 Package openai provides OpenAI API integration for the LLM gateway.
Package vllm provides vLLM OpenAI-compatible API integration for the LLM gateway.
 Package vllm provides vLLM OpenAI-compatible API integration for the LLM gateway.
Package xai provides xAI (Grok) API integration for the LLM gateway.
 Package xai provides xAI (Grok) API integration for the LLM gateway.
Package zai provides Z.ai API integration for the LLM gateway.
 Package zai provides Z.ai API integration for the LLM gateway.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.