eyrie

Documentation

Overview

Pluggable distributed cache backend interface.

CacheBackend abstracts the storage layer behind the in-process cache so that callers can swap in a distributed store (e.g. Redis) without changing the surrounding code. The default is an in-memory implementation; callers that do not opt in keep the existing in-process behavior unchanged.

Everything here is stdlib-only. The Redis backend speaks RESP directly over a net.Conn rather than pulling in a redis client dependency.

Index

• Constants
• Variables
• type CacheBackend
• type CacheStats
• type CacheWarmer
  • func NewCacheWarmer(chatFn func(ctx context.Context, messages []Message, opts ChatOptions) error, ...) *CacheWarmer
  • func (cw *CacheWarmer) CacheBreakpoints(systemPrompt string, conversationPrefix []Message) []int
  • func (cw *CacheWarmer) EstimateSavings(inputTokens int, requestCount int) float64
  • func (cw *CacheWarmer) ShouldWarm() bool
  • func (cw *CacheWarmer) Start(ctx context.Context) error
  • func (cw *CacheWarmer) Stop()
  • func (cw *CacheWarmer) Warm(ctx context.Context) error
• type ChatOptions
• type MemoryBackend
  • func NewMemoryBackend() *MemoryBackend
  • func (m *MemoryBackend) Delete(_ context.Context, key string) error
  • func (m *MemoryBackend) Get(_ context.Context, key string) ([]byte, bool, error)
  • func (m *MemoryBackend) Set(_ context.Context, key string, val []byte, ttl time.Duration) error
• type Message
• type RedisBackend
  • func NewRedisBackend(addr string, timeout time.Duration) *RedisBackend
  • func (r *RedisBackend) Close() error
  • func (r *RedisBackend) Delete(ctx context.Context, key string) error
  • func (r *RedisBackend) Get(ctx context.Context, key string) ([]byte, bool, error)
  • func (r *RedisBackend) Set(ctx context.Context, key string, val []byte, ttl time.Duration) error

Constants

const (
	// DefaultWarmInterval is the default interval between cache warming pings.
	// Anthropic's prompt cache TTL is 5 minutes; we warm at 4 minutes to stay ahead.
	DefaultWarmInterval = 4 * time.Minute
)

Default cache warmer settings.

Variables

var ErrRedisNil = errors.New("eyrie: redis nil reply")

ErrRedisNil is returned by the RESP client for a nil bulk-string reply (i.e. a missing key). It is handled internally and never surfaced from Get.

Types

type CacheBackend

type CacheBackend interface {
	// Get returns the value for key. The second return value reports whether the
	// key was present (and unexpired). A miss is (nil, false, nil), not an error.
	Get(ctx context.Context, key string) ([]byte, bool, error)
	// Set stores val under key. A non-positive ttl means "no expiry".
	Set(ctx context.Context, key string, val []byte, ttl time.Duration) error
	// Delete removes key. Deleting a missing key is not an error.
	Delete(ctx context.Context, key string) error
}

CacheBackend is the storage abstraction used by the cache. Implementations must be safe for concurrent use by multiple goroutines.

type CacheStats

type CacheStats struct {
	WarmingRequests     int64
	CacheHits           int64
	CacheMisses         int64
	EstimatedSavingsUSD float64
	LastWarmedAt        time.Time
}

CacheStats tracks cache warming statistics.

type CacheWarmer

type CacheWarmer struct {
	Provider     string
	Model        string
	SystemPrompt string
	Interval     time.Duration
	Enabled      bool

	Stats CacheStats

	// ChatFn is the function used to send warming requests.
	// It should send a chat completion request to the provider.
	ChatFn func(ctx context.Context, messages []Message, opts ChatOptions) error
	// contains filtered or unexported fields
}

CacheWarmer keeps Anthropic's prompt cache warm by periodically sending minimal requests with the system prompt, ensuring subsequent real requests get cache hits at a 90% discount.

func NewCacheWarmer

func NewCacheWarmer(chatFn func(ctx context.Context, messages []Message, opts ChatOptions) error, systemPrompt, provider, model string) *CacheWarmer

NewCacheWarmer creates a new CacheWarmer configured for the given provider. The chatFn is called to send warming pings; it should make a real API call.

func (*CacheWarmer) CacheBreakpoints

func (cw *CacheWarmer) CacheBreakpoints(systemPrompt string, conversationPrefix []Message) []int

CacheBreakpoints suggests where to place cache breakpoints in a message list. Anthropic allows up to 4 breakpoints. The strategy is:

• After system prompt (index 0 in returned slice signals "system prompt")
• After the first user message
• After large context blocks (messages with content > 200 chars)

Returns indices into the messages array where breakpoints should be placed. The special index -1 indicates the system prompt should be cached.

func (*CacheWarmer) EstimateSavings

func (cw *CacheWarmer) EstimateSavings(inputTokens int, requestCount int) float64

EstimateSavings calculates the cost savings from prompt caching in USD.

Without cache: inputTokens * requestCount * price_per_token With cache: (inputTokens * price_per_token * 1.25 first time) +

(inputTokens * (requestCount-1) * price_per_token * 0.1)

Returns the difference (savings amount).

func (*CacheWarmer) ShouldWarm

func (cw *CacheWarmer) ShouldWarm() bool

ShouldWarm returns true if the cache has likely expired (more than 4 minutes since the last warming request) and should be refreshed.

func (*CacheWarmer) Start

func (cw *CacheWarmer) Start(ctx context.Context) error

Start begins the background cache warming loop. It sends a warming ping immediately, then repeats every Interval until Stop is called or the context is cancelled. For non-Anthropic providers this is a no-op.

func (*CacheWarmer) Stop

func (cw *CacheWarmer) Stop()

Stop stops the cache warmer background loop.

func (*CacheWarmer) Warm

func (cw *CacheWarmer) Warm(ctx context.Context) error

Warm sends a single warming request immediately. This keeps the system prompt cached on Anthropic's side. Returns an error if the request fails. For non-Anthropic providers this is a no-op.

type ChatOptions

type ChatOptions struct {
	Model         string
	MaxTokens     int
	System        string
	EnableCaching bool
}

ChatOptions is a minimal options struct for cache warmer use.

type MemoryBackend

type MemoryBackend struct {
	// contains filtered or unexported fields
}

MemoryBackend is the default in-process CacheBackend. It stores values in a map guarded by a mutex and lazily evicts expired entries on access.

func NewMemoryBackend

func NewMemoryBackend() *MemoryBackend

NewMemoryBackend creates an empty in-memory CacheBackend.

func (*MemoryBackend) Delete

func (m *MemoryBackend) Delete(_ context.Context, key string) error

Delete implements CacheBackend.

func (*MemoryBackend) Get

func (m *MemoryBackend) Get(_ context.Context, key string) ([]byte, bool, error)

Get implements CacheBackend.

func (*MemoryBackend) Set

func (m *MemoryBackend) Set(_ context.Context, key string, val []byte, ttl time.Duration) error

Set implements CacheBackend.

type Message

type Message struct {
	Role    string
	Content string
}

Message is a minimal message struct for cache warmer use.

type RedisBackend

type RedisBackend struct {
	// contains filtered or unexported fields
}

RedisBackend is a minimal, dependency-free CacheBackend backed by Redis.

It implements only the subset of commands the cache needs:

GET <key>
SET <key> <val> [PX <ttl-ms>]
DEL <key>

(EXPIRE is expressed via SET ... PX rather than a separate command.)

The client speaks RESP over a single net.Conn protected by a mutex. This is a SKELETON: it is intentionally simple (one connection, no pooling, no pipelining, no AUTH/SELECT) and is meant as a starting point. For production use, extend with connection pooling, AUTH/db selection, and automatic reconnection.

func NewRedisBackend

func NewRedisBackend(addr string, timeout time.Duration) *RedisBackend

NewRedisBackend creates a RedisBackend targeting addr (host:port). The connection is established lazily on first use. A non-positive timeout falls back to a 5s default applied per operation.

func (*RedisBackend) Close

func (r *RedisBackend) Close() error

Close closes the underlying connection, if any.

func (*RedisBackend) Delete

func (r *RedisBackend) Delete(ctx context.Context, key string) error

Delete implements CacheBackend.

func (*RedisBackend) Get

func (r *RedisBackend) Get(ctx context.Context, key string) ([]byte, bool, error)

Get implements CacheBackend.

func (*RedisBackend) Set

func (r *RedisBackend) Set(ctx context.Context, key string, val []byte, ttl time.Duration) error

Set implements CacheBackend. A positive ttl is sent as a PX (millisecond) expiry.