embeddings

Documentation

Index

• type Embedder
• type EmbeddingCachedProvider
  • func NewEmbeddingCachedProvider(inner core.Provider, embedder Embedder, cfg SemanticCacheConfig) *EmbeddingCachedProvider
  • func (sp *EmbeddingCachedProvider) Chat(ctx context.Context, messages []core.GraycodeRouterMessage, ...) (*core.GraycodeRouterResponse, error)
  • func (sp *EmbeddingCachedProvider) Name() string
  • func (sp *EmbeddingCachedProvider) Ping(ctx context.Context) error
  • func (sp *EmbeddingCachedProvider) Stats() SemanticCacheStats
  • func (sp *EmbeddingCachedProvider) StreamChat(ctx context.Context, messages []core.GraycodeRouterMessage, ...) (*core.StreamResult, error)
• type EmbeddingParams
  • func DefaultEmbeddingParams(model string) EmbeddingParams
• type EmbeddingRequest
• type EmbeddingResponse
• type SemanticCacheConfig
  • func DefaultSemanticCacheConfig() SemanticCacheConfig
• type SemanticCacheStats

Types

type Embedder

type Embedder = core.Embedder

Embedder is the interface for creating embeddings.

type EmbeddingCachedProvider

type EmbeddingCachedProvider struct {
	// contains filtered or unexported fields
}

EmbeddingCachedProvider wraps a core.Provider and serves cached responses when a new request is semantically similar (cosine similarity above a threshold) to a previously seen request. Unlike CachedProvider's exact-match SHA256 keying, this tolerates paraphrases and minor wording changes.

It is safe for concurrent use. StreamChat is passed through unchanged.

func NewEmbeddingCachedProvider

func NewEmbeddingCachedProvider(inner core.Provider, embedder Embedder, cfg SemanticCacheConfig) *EmbeddingCachedProvider

NewEmbeddingCachedProvider wraps inner with a semantic (embedding-similarity) cache. embedder is used to embed requests; cfg zero-value fields are replaced with defaults.

func (*EmbeddingCachedProvider) Chat

func (sp *EmbeddingCachedProvider) Chat(ctx context.Context, messages []core.GraycodeRouterMessage, opts core.ChatOptions) (*core.GraycodeRouterResponse, error)

Chat returns a semantically-cached response on a hit; otherwise calls the inner provider and caches the result.

func (*EmbeddingCachedProvider) Name

func (sp *EmbeddingCachedProvider) Name() string

Name returns the inner provider's name.

func (*EmbeddingCachedProvider) Ping

func (sp *EmbeddingCachedProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider.

func (*EmbeddingCachedProvider) Stats

func (sp *EmbeddingCachedProvider) Stats() SemanticCacheStats

Stats returns current cache statistics.

func (*EmbeddingCachedProvider) StreamChat

func (sp *EmbeddingCachedProvider) StreamChat(ctx context.Context, messages []core.GraycodeRouterMessage, opts core.ChatOptions) (*core.StreamResult, error)

StreamChat delegates to the inner provider without caching.

type EmbeddingParams

type EmbeddingParams = core.EmbeddingParams

EmbeddingParams holds asymmetric params for indexing vs query.

func DefaultEmbeddingParams

func DefaultEmbeddingParams(model string) EmbeddingParams

DefaultEmbeddingParams returns known-good asymmetric params for common embedding models.

type EmbeddingRequest

type EmbeddingRequest = core.EmbeddingRequest

EmbeddingRequest represents an embedding API call.

type EmbeddingResponse

type EmbeddingResponse = core.EmbeddingResponse

EmbeddingResponse holds embedding results.

type SemanticCacheConfig

type SemanticCacheConfig struct {
	// MaxAge is how long cache entries remain valid. Default: 5 minutes.
	MaxAge time.Duration
	// MaxSize is the maximum number of cached responses. Default: 100.
	// When exceeded, the least-recently-used entry is evicted.
	MaxSize int
	// Enabled toggles caching. Default: true.
	Enabled bool
	// TemperatureThreshold is the temperature above which responses are not
	// cached. Default: 0.5.
	TemperatureThreshold float64
	// SimilarityThreshold is the minimum cosine similarity (0..1) for a cached
	// entry to be considered a hit. Default: 0.95.
	SimilarityThreshold float64
	// EmbeddingModel is the model passed to the Embedder. Required.
	EmbeddingModel string
}

SemanticCacheConfig controls the behavior of EmbeddingCachedProvider.

func DefaultSemanticCacheConfig

func DefaultSemanticCacheConfig() SemanticCacheConfig

DefaultSemanticCacheConfig returns a SemanticCacheConfig with sensible defaults.

type SemanticCacheStats

type SemanticCacheStats struct {
	Size    int  `json:"size"`
	MaxSize int  `json:"max_size"`
	Hits    int  `json:"hits"`
	Misses  int  `json:"misses"`
	Enabled bool `json:"enabled"`
}

SemanticCacheStats reports cache occupancy and hit/miss counts.