cache

Documentation

Overview

Package cache provides a TTL + LRU response cache for LLM completions. Cache keys are derived from canonicalized request JSON, skipping non-deterministic fields (stream, user, request_id) and stripping timestamp prefixes from message content.

Index

• Constants
• func CacheKey(body []byte) (string, error)
• type Cache
  • func New(opts ...Option) *Cache
  • func (c *Cache) Clear()
  • func (c *Cache) Get(body []byte, noCache bool) (Entry, bool)
  • func (c *Cache) Len() int
  • func (c *Cache) Prune() int
  • func (c *Cache) Set(body []byte, entry Entry)
  • func (c *Cache) SetEnabled(b bool)
  • func (c *Cache) Stats() Stats
• type Entry
• type Option
  • func WithEnabled(b bool) Option
  • func WithMaxItemSize(n int) Option
  • func WithMaxSize(n int) Option
  • func WithTTL(d time.Duration) Option
• type Stats

Constants

const (
	// DefaultTTL is the default time-to-live for cached entries.
	DefaultTTL = 10 * time.Minute
	// DefaultMaxSize is the default maximum number of cached entries.
	DefaultMaxSize = 200
	// DefaultMaxItemSize is the maximum size of a single cached item (1 MB).
	DefaultMaxItemSize = 1 << 20
)

Functions

func CacheKey

func CacheKey(body []byte) (string, error)

CacheKey returns a hex-encoded SHA-256 hash of the canonicalized request JSON, omitting non-deterministic fields and stripping timestamps.

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a concurrency-safe TTL + LRU response cache.

func New

func New(opts ...Option) *Cache

New creates a Cache with the given options.

func (*Cache) Clear

func (c *Cache) Clear()

Clear removes all entries and resets stats.

func (*Cache) Get

func (c *Cache) Get(body []byte, noCache bool) (Entry, bool)

Get retrieves a cached response for the given request body. Returns the entry and true on hit, or a zero Entry and false on miss. If noCache is true (e.g. from a Cache-Control: no-cache header), it always returns a miss.

func (*Cache) Len

func (c *Cache) Len() int

Len returns the number of items currently in the cache.

func (*Cache) Prune

func (c *Cache) Prune() int

Prune removes expired entries and returns the count removed.

func (*Cache) Set

func (c *Cache) Set(body []byte, entry Entry)

Set stores a response in the cache, keyed by the request body.

func (*Cache) SetEnabled

func (c *Cache) SetEnabled(b bool)

SetEnabled enables or disables the cache at runtime.

func (*Cache) Stats

func (c *Cache) Stats() Stats

Stats returns cache performance statistics.

type Entry

type Entry struct {
	Body       []byte
	StatusCode int
	Header     map[string][]string
	CreatedAt  time.Time
}

Entry is a cached response.

type Option

type Option func(*Cache)

Option configures a Cache.

func WithEnabled

func WithEnabled(b bool) Option

WithEnabled sets whether the cache is active.

func WithMaxItemSize

func WithMaxItemSize(n int) Option

WithMaxItemSize sets the maximum byte size of a single cached item.

func WithMaxSize

func WithMaxSize(n int) Option

WithMaxSize sets the maximum number of cached entries.

func WithTTL

func WithTTL(d time.Duration) Option

WithTTL sets the cache entry TTL.

type Stats

type Stats struct {
	Hits      int64   `json:"hits"`
	Misses    int64   `json:"misses"`
	Evictions int64   `json:"evictions"`
	HitRate   float64 `json:"hitRate"`
}

Stats holds cache performance counters.