provider

Documentation

Overview

Package provider provides the building blocks to create LLM API clients.

Index

• Constants
• type ClientFunc
  • func WithDefaulModel(model string) ClientFunc
  • func WithEndpoint(endpoint string) ClientFunc
  • func WithToken(token string) ClientFunc
• type ClientOptions
• type Func
  • func WithMaxToken(maxToken int) Func
  • func WithModel(model string) Func
  • func WithResponseBody(requestBody any) Func
  • func WithSeed(seed int) Func
  • func WithStream(stream bool) Func
  • func WithSystemMessages(systemMessage ...string) Func
  • func WithTemperature(temperature float64) Func
  • func WithTopK(topK int) Func
  • func WithTopP(topP float64) Func
  • func WithUserMessages(userMessage ...string) Func
• type IMeta
• type IMetrics
• type IProvider
• type Map
  • func (m Map) Completion(ctx context.Context, options ...Func) (map[string]string, error)
  • func (m Map) String() string
  • func (m Map) ToSlice() []IProvider
• type Options
  • func NewOptionsFrom(options ...Func) (*Options, error)
• type Provider
  • func New(name string, options ...ClientFunc) (*Provider, error)
  • func (s *Provider) GetCounterCompletion() *expvar.Int
  • func (s *Provider) GetCounterCompletionFailed() *expvar.Int
  • func (s *Provider) GetCounterCounted() *expvar.Int
  • func (s *Provider) GetLogger() sypl.ISypl
  • func (s *Provider) GetName() string
  • func (s *Provider) GetType() string

Constants

const (
	Type = "provider"
)

Type is the type of the entity regarding the framework. It is used to for example, to identify the entity in the logs, metrics, and for tracing.

Types

type ClientFunc

type ClientFunc func(o *ClientOptions) error

ClientFunc allows to set provider options.

func WithDefaulModel

func WithDefaulModel(model string) ClientFunc

WithDefaulModel sets the default model.

func WithEndpoint

func WithEndpoint(endpoint string) ClientFunc

WithEndpoint sets the provider endpoint.

func WithToken

func WithToken(token string) ClientFunc

WithToken sets the provider token.

type ClientOptions

type ClientOptions struct {
	// Endpoint to reach the provider.
	Endpoint string `json:"endpoint" validate:"required"`

	// Model default model to be used.
	Model string `json:"model,omitempty"`

	// Token to authenticate against the provider.
	Token string `json:"-"`
}

ClientOptions for the provider.

type Func

type Func func(o *Options) error

Func allows to set options.

func WithMaxToken

func WithMaxToken(maxToken int) Func

WithMaxToken sets the maxToken option.

func WithModel

func WithModel(model string) Func

WithModel sets the model option.

func WithResponseBody

func WithResponseBody(requestBody any) Func

WithResponseBody sets the responseBody option.

func WithSeed

func WithSeed(seed int) Func

WithSeed sets the seed option.

func WithStream

func WithStream(stream bool) Func

WithStream sets the stream option.

func WithSystemMessages

func WithSystemMessages(systemMessage ...string) Func

WithSystemMessages sets the systemMessages option.

func WithTemperature

func WithTemperature(temperature float64) Func

WithTemperature sets the temperature option.

func WithTopK

func WithTopK(topK int) Func

WithTopK sets the topK option.

func WithTopP

func WithTopP(topP float64) Func

WithTopP sets the topP option.

func WithUserMessages

func WithUserMessages(userMessage ...string) Func

WithUserMessages sets the userMessages option.

type IMeta

type IMeta interface {
	GetClient() any

	// GetLogger returns the logger.
	GetLogger() sypl.ISypl

	// GetName returns the provider name.
	GetName() string

	// GetType returns its type.
	GetType() string
}

IMeta definition.

type IMetrics

type IMetrics interface {
	// GetCounterCompletion returns the completion metric.
	GetCounterCompletion() *expvar.Int

	// GetCounterCompletionFailed returns the failed completion metric.
	GetCounterCompletionFailed() *expvar.Int
}

IMetrics definition.

type IProvider

type IProvider interface {
	IMeta

	IMetrics

	// Completion generates a completion using the provider API.
	// Optionally pass WithResponseBody to unmarshal the response body.
	// It will always return the original, unparsed response body, if no error.
	//
	// NOTE: Not all options are available for all providers.
	Completion(ctx context.Context, options ...Func) (string, error)
}

IProvider defines what a provider does.

type Map

type Map map[string]IProvider

Map is a map of strgs.

func (Map) Completion

func (m Map) Completion(
	ctx context.Context,
	options ...Func,
) (map[string]string, error)

CompletionMany calls the Completion concurrently against all providers in the map. The response is mapped to the provider name.

func (Map) String

func (m Map) String() string

String implements the Stringer interface.

func (Map) ToSlice

func (m Map) ToSlice() []IProvider

ToSlice converts Map to Slice of IProvider.

type Options

type Options struct {
	// Model is the model to be used.
	Model string `json:"model" validate:"required"`

	// UserMessages is the user role messages.
	UserMessages []string `json:"userMessages" validate:"required"`

	// MaxTokens defines the max amount of tokens in the response. Default to 0
	// which means no limit.
	MaxTokens int `json:"maxToken,omitempty" validate:"gte=0"`

	// ResponseBody is the request body.
	ResponseBody any `json:"requestBody"`

	// Seed the LLM will make a best effort to sample deterministically, such
	// that repeated requests with the same seed and parameters should return
	// the same result. Default to not set which means no determinism.
	//
	// NOTE: Determinism is not guaranteed.
	Seed int `json:"seed,omitempty" validate:"gte=0"`

	// Stream determines if the response should be streamed or not. Default to
	// false which means not to stream.
	Stream bool `json:"stream"`

	// SystemMessages is the system role messages.
	SystemMessages []string `json:"systemMessages,omitempty"`

	// Temperature to use, between 0 and 2. Higher values like 0.8 will make the
	// output more random, while lower values like 0.2 will make it more focused
	// and deterministic. Default to 1.0 which means be creative.
	//
	// NOTE: It's generally recommended altering this OR temperature but not
	// both!
	Temperature float64 `json:"temperature,omitempty" validate:"gte=0"`

	// TopK means the number of highest probability vocabulary tokens to keep
	// for sampling, default is not set (0) which means no restrictions.
	TopK int `json:"topK,omitempty" validate:"gte=0"`

	// TopP is an alternative to sampling with temperature, called nucleus
	// sampling, where the model considers the results of the tokens with top_p
	// probability mass. So 0.1 means only the tokens comprising the top 10%
	// probability mass are considered. Default to 1.0 which means consider all
	// tokens.
	//
	// NOTE: It's generally recommended altering this OR temperature but not
	// both!
	TopP float64 `json:"topP,omitempty" validate:"gte=0"`
}

Options for operations.

func NewOptionsFrom

func NewOptionsFrom(options ...Func) (*Options, error)

NewOptionsFrom process, and validate against the default options.

type Provider

type Provider struct {
	// Logger.
	Logger sypl.ISypl `json:"-" validate:"required"`

	// Name of the provider type.
	Name string `json:"name" validate:"required,lowercase,gte=1"`

	// A provider may have the following...
	// Endpoint to reach the provider.
	Endpoint string `json:"endpoint,omitempty"`

	// DefaultModel default model to be used.
	DefaultModel string `json:"model,omitempty"`

	// Token to authenticate against the provider.
	Token string `json:"-"`
	// contains filtered or unexported fields
}

Provider definition.

func New

func New(
	name string,
	options ...ClientFunc,
) (*Provider, error)

New returns a new provider.

func (*Provider) GetCounterCompletion

func (s *Provider) GetCounterCompletion() *expvar.Int

GetCounterCompletion returns the completion metric.

func (*Provider) GetCounterCompletionFailed

func (s *Provider) GetCounterCompletionFailed() *expvar.Int

GetCounterCompletionFailed returns the failed completion metric.

func (*Provider) GetCounterCounted

func (s *Provider) GetCounterCounted() *expvar.Int

GetCounterCounted returns the metric.

func (*Provider) GetLogger

func (s *Provider) GetLogger() sypl.ISypl

GetLogger returns the logger.

func (*Provider) GetName

func (s *Provider) GetName() string

GetName returns the provider name.

func (*Provider) GetType

func (s *Provider) GetType() string

GetType returns its type.