cache

Documentation

Overview

Package cache implements semantic response caching backed by NuclaDB: it embeds the request's prompt text, looks up the nearest cached vector, and serves the stored response verbatim on a close-enough match. Both non-streaming and streaming (SSE) responses are cached; a streamed hit is replayed as one write of the originally captured bytes rather than re-created chunk by chunk.

The embedding model's output dimensionality must match the dimension the target NuclaDB instance was started with (nucladbd -dim); this package doesn't validate that, NuclaDB's insert call will just reject mismatched vectors.

NuclaDB's cosine-metric "score" is a distance (1 - cosine similarity), not a similarity: 0 means identical, larger means further apart. MaxDistance is compared against that distance directly — don't treat it as a similarity threshold.

Index

• func PromptText(messages []Message) string
• type Cache
  • func New(cfg Config) *Cache
  • func (c *Cache) EnsureTenant(ctx context.Context) error
  • func (c *Cache) Lookup(ctx context.Context, text string) (body []byte, contentType string, hit bool, err error)
  • func (c *Cache) Store(ctx context.Context, text string, response []byte, contentType string) error
• type Config
• type Message

Functions

func PromptText

func PromptText(messages []Message) string

PromptText joins the content of chat messages into the text that gets embedded and used as the cache key.

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

func New

func New(cfg Config) *Cache

func (*Cache) EnsureTenant

func (c *Cache) EnsureTenant(ctx context.Context) error

EnsureTenant creates the cache's NuclaDB tenant. It's safe to call every startup: an "already exists" error from NuclaDB is not surfaced.

func (*Cache) Lookup

func (c *Cache) Lookup(ctx context.Context, text string) (body []byte, contentType string, hit bool, err error)

Lookup embeds text and returns the cached response body (and the content type it was originally served with — text/event-stream for a streamed response, application/json otherwise) for the nearest match if its distance is within MaxDistance.

func (*Cache) Store

func (c *Cache) Store(ctx context.Context, text string, response []byte, contentType string) error

Store embeds text and upserts it into NuclaDB with response and contentType attached as metadata, keyed by a hash of text so identical prompts overwrite in place.

type Config

type Config struct {
	NuclaDBAddr          string
	EmbeddingBackendAddr string
	EmbeddingModel       string
	MaxDistance          float32
	TenantID             string
}

type Message

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}