cache

Documentation

Overview

Package cache wraps an OpenAI-compatible client with a response cache.

The cache backend is supplied by the caller via Cacher (which is just the github.com/rakunlabs/cache.Cacher interface, re-exported for convenience). This means callers pick their own storage — in-memory LRU, Redis, or anything else that implements the three-method interface — without forcing a dependency on this package.

Typical use:

import (
    cachelib "github.com/rakunlabs/cache"
    "github.com/rakunlabs/cache/store/memory"
)

store, _ := cachelib.New[string, []byte](ctx, memory.Store,
    cachelib.WithStoreConfig(&memory.Config{MaxItems: 500, TTL: 10*time.Minute}),
)
cached := cache.New(client, store)
resp, err := cached.Chat(ctx, &agent.ChatRequest{...})

Index

• func DefaultKeyer(req *agent.ChatRequest) string
• func HitMarker(resp *agent.ChatResponse) bool
• type Cache
  • func New(inner ChatClient, store Cacher, opts ...Option) *Cache
  • func (c *Cache) Chat(ctx context.Context, req *agent.ChatRequest) (*agent.ChatResponse, error)
  • func (c *Cache) ChatStream(ctx context.Context, req *agent.ChatRequest) (*agent.Stream, error)
  • func (c *Cache) Embeddings(ctx context.Context, req *agent.EmbeddingRequest) (*agent.EmbeddingResponse, error)
• type Cacher
• type ChatClient
• type Keyer
• type Option
  • func WithKeyer(k Keyer) Option
  • func WithNegativeCache(on bool) Option
  • func WithOnHit(f func(req *agent.ChatRequest)) Option
  • func WithOnMiss(f func(req *agent.ChatRequest)) Option
  • func WithSkipWhen(f func(*agent.ChatRequest) bool) Option
  • func WithStreams(on bool) Option
• type Options

Functions

func DefaultKeyer

func DefaultKeyer(req *agent.ChatRequest) string

DefaultKeyer hashes the canonical JSON of the request. Map iteration order is normalised so callers don't have to worry about it.

The key includes: model, messages, tools, tool_choice, temperature, top_p, seed, response_format, max_tokens, max_completion_tokens, reasoning_effort, parallel_tool_calls, stop, presence/frequency penalty, logit_bias, user, and the entire Extra map. It deliberately ignores stream / stream_options because that's a transport concern, not a content concern.

func HitMarker

func HitMarker(resp *agent.ChatResponse) bool

HitMarker returns true if resp looks like a cache hit emitted by *Cache. Useful for tests and metrics. False for nil / no-usage responses.

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache wraps a ChatClient with a request-keyed response cache.

func New

func New(inner ChatClient, store Cacher, opts ...Option) *Cache

New constructs a Cache backed by store. store must be the Cacher from github.com/rakunlabs/cache (typically a *cache.Cache returned by cache.New).

func (*Cache) Chat

func (c *Cache) Chat(ctx context.Context, req *agent.ChatRequest) (*agent.ChatResponse, error)

Chat looks up a cached response, or calls the underlying client and stores the result on a miss. Errors from the inner client are returned as-is and (by default) not cached.

func (*Cache) ChatStream

func (c *Cache) ChatStream(ctx context.Context, req *agent.ChatRequest) (*agent.Stream, error)

ChatStream optionally records and replays the underlying stream. When WithStreams(false) (the default), it is a pure passthrough so the caller always sees fresh upstream pacing.

func (*Cache) Embeddings

func (c *Cache) Embeddings(ctx context.Context, req *agent.EmbeddingRequest) (*agent.EmbeddingResponse, error)

Embeddings is a passthrough today. We could cache deterministic-input embeddings in v2 — most callers already cache vectors at a higher layer.

type Cacher

type Cacher = cache.Cacher[string, []byte]

Cacher is the storage interface the wrapper needs. It is identical to github.com/rakunlabs/cache.Cacher re-exported here so callers don't have to import the upstream package just to reference the type. Both *cache.Cache and the underlying cache.Cacher satisfy it.

type ChatClient

type ChatClient interface {
	Chat(ctx context.Context, req *agent.ChatRequest) (*agent.ChatResponse, error)
	ChatStream(ctx context.Context, req *agent.ChatRequest) (*agent.Stream, error)
	Embeddings(ctx context.Context, req *agent.EmbeddingRequest) (*agent.EmbeddingResponse, error)
}

ChatClient mirrors the minimal interface of *agent.Client that the cache wrapper depends on. Any wrapper (e.g. governance) that exposes the same three methods works in its place.

type Keyer

type Keyer func(*agent.ChatRequest) string

Keyer turns a *agent.ChatRequest into a stable cache key. Implementations must be deterministic — two equivalent requests must hash to the same key or hit-rate will be zero.

type Option

type Option func(*Options)

Option mutates Options.

func WithKeyer

func WithKeyer(k Keyer) Option

WithKeyer overrides the default request canonicaliser.

func WithNegativeCache

func WithNegativeCache(on bool) Option

WithNegativeCache controls whether 4xx-class results are cached. Default is off so transient mistakes don't poison the cache.

func WithOnHit

func WithOnHit(f func(req *agent.ChatRequest)) Option

WithOnHit installs a callback fired on every cache hit (after read, before returning the response). Useful for metrics.

func WithOnMiss

func WithOnMiss(f func(req *agent.ChatRequest)) Option

WithOnMiss is the dual of WithOnHit.

func WithSkipWhen

func WithSkipWhen(f func(*agent.ChatRequest) bool) Option

WithSkipWhen installs a predicate that, when it returns true, bypasses the cache entirely (no read, no write). The default skip rule is:

temperature > 0 && seed == nil

— i.e. only deterministic-looking requests are cached.

func WithStreams

func WithStreams(on bool) Option

WithStreams enables stream record/replay. Default is off because a replayed stream loses the original wall-clock pacing.

type Options

type Options struct {
	Keyer    Keyer
	SkipWhen func(*agent.ChatRequest) bool
	Streams  bool
	Negative bool
	OnHit    func(req *agent.ChatRequest)
	OnMiss   func(req *agent.ChatRequest)
}

Options holds cache configuration.