llmresolver

Documentation

Index

• Constants
• Variables
• func Chat(ctx context.Context, req Request, ...) (libmodelprovider.LLMChatClient, libmodelprovider.Provider, string, error)
• func Embed(ctx context.Context, embedReq EmbedRequest, ...) (libmodelprovider.LLMEmbedClient, libmodelprovider.Provider, string, error)
• func HighestContext(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error)
• func NormalizeModelName(modelName string) string
• func PolicyFromString(name string) (...)
• func PromptExecute(ctx context.Context, req Request, ...) (libmodelprovider.LLMPromptExecClient, libmodelprovider.Provider, string, error)
• func Randomly(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error)
• func Stream(ctx context.Context, req Request, ...) (libmodelprovider.LLMStreamClient, libmodelprovider.Provider, string, error)
• type EmbedRequest
• type Request
• type RequestResolver
  • func (r *RequestResolver) ResolveChat(ctx context.Context, req Request) (libmodelprovider.LLMChatClient, libmodelprovider.Provider, string, error)
  • func (r *RequestResolver) ResolveEmbed(ctx context.Context, req EmbedRequest) (libmodelprovider.LLMEmbedClient, libmodelprovider.Provider, string, error)
  • func (r *RequestResolver) ResolvePromptExecute(ctx context.Context, req Request) (libmodelprovider.LLMPromptExecClient, libmodelprovider.Provider, string, error)
  • func (r *RequestResolver) ResolveStream(ctx context.Context, req Request) (libmodelprovider.LLMStreamClient, libmodelprovider.Provider, string, error)
• type Resolver
  • func NewRequestResolver(...) Resolver

Constants

const (
	StrategyRandom      = "random"
	StrategyAuto        = "auto"
	StrategyLowLatency  = "low-latency"
	StrategyLowPriority = "low-prio"
)

Variables

var ErrNoAvailableModels = errors.New("no models found in runtime state")

ErrNoAvailableModels is returned when no providers are available.

var ErrNoSatisfactoryModel = errors.New("no model matched the requirements")

ErrNoSatisfactoryModel is returned when providers exist but none match requirements.

Functions

func Chat

func Chat(
	ctx context.Context,
	req Request,
	getModels func(ctx context.Context, backendTypes ...string) ([]libmodelprovider.Provider, error),
	resolver func(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error),
) (libmodelprovider.LLMChatClient, libmodelprovider.Provider, string, error)

Chat implements the chat resolution workflow using the provided dependencies.

func Embed

func Embed(
	ctx context.Context,
	embedReq EmbedRequest,
	getModels func(ctx context.Context, backendTypes ...string) ([]libmodelprovider.Provider, error),
	resolver func(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error),
) (libmodelprovider.LLMEmbedClient, libmodelprovider.Provider, string, error)

Embed implements the embedding resolution workflow using the provided dependencies.

func HighestContext

func HighestContext(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error)

HighestContext is a policy that selects the provider with the largest context window.

When multiple providers have the same context length, one is chosen randomly. This is useful for tasks requiring long context windows.

func NormalizeModelName

func NormalizeModelName(modelName string) string

NormalizeModelName standardizes model names for comparison

func PolicyFromString

func PolicyFromString(name string) (func(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error), error)

PolicyFromString maps string names to resolver policies

func PromptExecute

func PromptExecute(
	ctx context.Context,
	req Request,
	getModels func(ctx context.Context, backendTypes ...string) ([]libmodelprovider.Provider, error),
	resolver func(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error),
) (libmodelprovider.LLMPromptExecClient, libmodelprovider.Provider, string, error)

PromptExecute implements the prompt execution resolution workflow using the provided dependencies.

func Randomly

func Randomly(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error)

Randomly is a policy that selects a random provider and random backend.

This provides basic load balancing across available resources.

func Stream

func Stream(
	ctx context.Context,
	req Request,
	getModels func(ctx context.Context, backendTypes ...string) ([]libmodelprovider.Provider, error),
	resolver func(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error),
) (libmodelprovider.LLMStreamClient, libmodelprovider.Provider, string, error)

Stream implements the streaming resolution workflow using the provided dependencies.

Types

type EmbedRequest

type EmbedRequest struct {
	// ModelName specifies the preferred embedding model.
	// If empty, a default model will be selected.
	ModelName string

	// ProviderType specifies which provider implementation to use.
	// If empty, any provider is acceptable.
	ProviderType string

	// Tracker is used for activity monitoring and tracing.
	Tracker libtracker.ActivityTracker
}

EmbedRequest is a specialized request for embedding operations.

type Request

type Request struct {
	// ProviderTypes specifies which provider implementations to consider.
	// If empty, all available providers are considered.
	ProviderTypes []string

	// ModelNames specifies preferred model names in priority order.
	// The resolver will try these models first before considering others.
	// If empty, any model is acceptable.
	ModelNames []string

	// ContextLength specifies the minimum required context window length.
	// Providers with smaller context windows will be excluded.
	// If 0, no minimum is enforced.
	ContextLength int

	// Tracker is used for activity monitoring and tracing.
	// While not serializable, it's preserved through resolution chains.
	Tracker libtracker.ActivityTracker
}

Request contains requirements for selecting a model provider.

type RequestResolver

type RequestResolver struct {
	// contains filtered or unexported fields
}

func (*RequestResolver) ResolveChat

func (r *RequestResolver) ResolveChat(ctx context.Context, req Request) (libmodelprovider.LLMChatClient, libmodelprovider.Provider, string, error)

ResolveChat implements Resolver by using the struct's getModels and resolver fields.

func (*RequestResolver) ResolveEmbed

func (r *RequestResolver) ResolveEmbed(ctx context.Context, req EmbedRequest) (libmodelprovider.LLMEmbedClient, libmodelprovider.Provider, string, error)

ResolveEmbed implements Resolver by using the struct's getModels and resolver fields.

func (*RequestResolver) ResolvePromptExecute

func (r *RequestResolver) ResolvePromptExecute(ctx context.Context, req Request) (libmodelprovider.LLMPromptExecClient, libmodelprovider.Provider, string, error)

ResolvePromptExecute implements Resolver by using the struct's getModels and resolver fields.

func (*RequestResolver) ResolveStream

func (r *RequestResolver) ResolveStream(ctx context.Context, req Request) (libmodelprovider.LLMStreamClient, libmodelprovider.Provider, string, error)

ResolveStream implements Resolver by using the struct's getModels and resolver fields.

type Resolver

type Resolver interface {
	// ResolveChat selects a provider capable of handling chat requests
	// and returns a connected client for that provider.
	//
	// Parameters:
	//   ctx: Context for cancellation and timeouts
	//   req: Requirements for the model (provider types, model names, context length)
	//   getProviders: Function to fetch available providers (decoupled from storage)
	//   policy: Selection strategy for choosing among candidates
	//
	// Returns:
	//   client: Ready-to-use chat client
	//   provider: The selected provider metadata
	//   backendID: Identifier for the specific backend instance
	//   error: Any error encountered during resolution
	ResolveChat(
		ctx context.Context,
		req Request) (modelrepo.LLMChatClient, modelrepo.Provider, string, error)

	// ResolvePromptExecute selects a provider capable of executing prompt-based requests
	// and returns a connected client for that provider.
	//
	// See ResolveChat for parameter and return details.
	ResolvePromptExecute(
		ctx context.Context,
		req Request,
	) (modelrepo.LLMPromptExecClient, modelrepo.Provider, string, error)

	// ResolveEmbed selects a provider capable of generating embeddings
	// and returns a connected client for that provider.
	//
	// Parameters:
	//   ctx: Context for cancellation and timeouts
	//   req: Requirements for the embedding model
	//   getProviders: Function to fetch available providers
	//   policy: Selection strategy for choosing among candidates
	//
	// Returns:
	//   client: Ready-to-use embedding client
	//   provider: The selected provider metadata
	//   backendID: Identifier for the specific backend instance
	//   error: Any error encountered during resolution
	ResolveEmbed(
		ctx context.Context,
		req EmbedRequest,
	) (modelrepo.LLMEmbedClient, modelrepo.Provider, string, error)

	// ResolveStream selects a provider capable of streaming responses
	// and returns a connected client for that provider.
	//
	// See ResolveEmbed for parameter and return details.
	ResolveStream(
		ctx context.Context,
		req Request,
	) (modelrepo.LLMStreamClient, modelrepo.Provider, string, error)
}

Resolver is the interface for selecting appropriate model providers based on requirements.

The resolver handles the process of: 1. Finding available providers that match requirements 2. Selecting the best provider/backend combination 3. Establishing a connection to the selected backend

func NewRequestResolver

func NewRequestResolver(
	getModels func(ctx context.Context, backendTypes ...string) ([]libmodelprovider.Provider, error),
	resolver func(candidates []libmodelprovider.Provider) (libmodelprovider.Provider, string, error),
) Resolver

NewRequestResolver creates a new Resolver implementation with the specified dependencies. This is the preferred way to instantiate a resolver.