cache

Documentation

Overview

Package cache wraps a types.Embedder with a hot-path LRU cache so repeated embed calls on the same text return without paying the model cost.

Use case: the agent issues semantic_search with the same intent multiple times during a multi-hop flow, and explain_match re-embeds the intent to compare each candidate. The cache keeps both hot.

The cache is intentionally in-process and not persisted — the embeddings depend on model identity, which changes when the user swaps backends. Restart clears the cache, which is the correct behavior.

Index

• Constants
• type Cached
  • func New(emb types.Embedder, size int) *Cached
  • func (c *Cached) Dimension() int
  • func (c *Cached) Embed(ctx context.Context, batch []string) ([][]float32, error)
  • func (c *Cached) MaxInputTokens() int
  • func (c *Cached) Name() string
  • func (c *Cached) Stats() (hits, miss, size int)

Constants

const DefaultSize = 1024

DefaultSize is the cache capacity when the caller does not specify. 1024 entries × 1024-dim × 4 bytes ≈ 4 MB per cache instance for the stored vectors plus a small per-entry book-keeping overhead.

Types

type Cached

type Cached struct {
	// contains filtered or unexported fields
}

Cached wraps an inner Embedder. Cache hits return immediately from memory; misses pass through to inner.Embed and store the result.

func New

func New(emb types.Embedder, size int) *Cached

New wraps emb with an LRU cache. size <= 0 uses DefaultSize.

func (*Cached) Dimension

func (c *Cached) Dimension() int

func (*Cached) Embed

func (c *Cached) Embed(ctx context.Context, batch []string) ([][]float32, error)

Embed checks the cache for each batch entry, calls the inner embedder only for misses, and stores all results before returning.

func (*Cached) MaxInputTokens

func (c *Cached) MaxInputTokens() int

func (*Cached) Name

func (c *Cached) Name() string

Name / Dimension / MaxInputTokens forward to the inner embedder so the wrapper is a drop-in for types.Embedder.

func (*Cached) Stats

func (c *Cached) Stats() (hits, miss, size int)

Stats returns hit / miss counters. Useful for /health diagnostics and tests asserting the cache is being exercised.