cache

Documentation

Index

• func BuildCacheKey(messages []RelayMessage, opts ChatOptions) string
• type AnthropicCachedMessage
  • func AddCacheBreakpoints(messages []RelayMessage) []AnthropicCachedMessage
• type CacheConfig
  • func DefaultCacheConfig() CacheConfig
• type CacheControlType
• type CacheStatsResult
• type CachedContent
• type CachedProvider
  • func NewCachedProvider(inner Provider, cfg CacheConfig) *CachedProvider
  • func (cp *CachedProvider) CacheStats() CacheStatsResult
  • func (cp *CachedProvider) Chat(ctx context.Context, messages []RelayMessage, opts ChatOptions) (*RelayResponse, error)
  • func (cp *CachedProvider) ClearCache()
  • func (cp *CachedProvider) Name() string
  • func (cp *CachedProvider) Ping(ctx context.Context) error
  • func (cp *CachedProvider) SetEnabled(enabled bool)
  • func (cp *CachedProvider) StreamChat(ctx context.Context, messages []RelayMessage, opts ChatOptions) (*StreamResult, error)
• type ChatOptions
• type Provider
• type RelayMessage
• type RelayResponse
• type StreamResult

Functions

func BuildCacheKey

func BuildCacheKey(messages []RelayMessage, opts ChatOptions) string

BuildCacheKey returns the deterministic key used by the response cache, or "" when the request cannot be cached.

Types

type AnthropicCachedMessage

type AnthropicCachedMessage struct {
	Role         string      `json:"role"`
	Content      interface{} `json:"content"` // string or []CachedContent
	CacheControl interface{} `json:"cache_control,omitempty"`
}

AnthropicCachedMessage is an Anthropic message with optional cache_control.

func AddCacheBreakpoints

func AddCacheBreakpoints(messages []RelayMessage) []AnthropicCachedMessage

AddCacheBreakpoints returns a copy of messages with Anthropic cache_control breakpoints applied following the recommended pattern:

• Breakpoint on the second-to-last message (caches conversation prefix)
• The last user message is left uncached (always new)

Only applies to messages with role "user" or "assistant". No-op if fewer than 2 messages.

type CacheConfig

type CacheConfig struct {
	// MaxAge is how long cache entries remain valid. Default: 5 minutes.
	MaxAge time.Duration
	// MaxSize is the maximum number of cached responses. Default: 100.
	// When exceeded, the least-recently-used entry is evicted.
	MaxSize int
	// Enabled toggles caching. Default: true.
	// When false, the CachedProvider passes all requests through unchanged.
	Enabled bool
	// TemperatureThreshold is the temperature above which responses are not cached.
	// Default: 0.5. Responses with temperature > threshold are expected to vary,
	// so caching them would defeat the purpose.
	TemperatureThreshold float64
}

CacheConfig controls the behavior of CachedProvider.

func DefaultCacheConfig

func DefaultCacheConfig() CacheConfig

DefaultCacheConfig returns a CacheConfig with sensible defaults.

type CacheControlType

type CacheControlType string

CacheControlType is the type of cache control.

const (
	// CacheControlEphemeral caches content for up to 5 minutes.
	CacheControlEphemeral CacheControlType = "ephemeral"
)

type CacheStatsResult

type CacheStatsResult struct {
	Size    int  `json:"size"`
	MaxSize int  `json:"max_size"`
	Enabled bool `json:"enabled"`
}

CacheStatsResult holds cache statistics.

type CachedContent

type CachedContent struct {
	Type         string      `json:"type"`
	Text         string      `json:"text"`
	CacheControl interface{} `json:"cache_control,omitempty"`
}

CachedContent wraps a content string with cache control metadata.

type CachedProvider

type CachedProvider struct {
	// contains filtered or unexported fields
}

CachedProvider wraps a Provider and caches non-streaming responses based on a hash of the input parameters. Inspired by maximhq/bifrost's caching layer.

CachedProvider is safe for concurrent use.

func NewCachedProvider

func NewCachedProvider(inner Provider, cfg CacheConfig) *CachedProvider

NewCachedProvider wraps inner with a response cache configured by cfg. Zero-value fields in cfg are replaced with defaults.

func (*CachedProvider) CacheStats

func (cp *CachedProvider) CacheStats() CacheStatsResult

CacheStats returns the current number of entries in the cache.

func (*CachedProvider) Chat

func (cp *CachedProvider) Chat(ctx context.Context, messages []RelayMessage, opts ChatOptions) (*RelayResponse, error)

Chat checks the cache first. On a miss, it calls the inner provider and caches the response (if the temperature is not too high).

func (*CachedProvider) ClearCache

func (cp *CachedProvider) ClearCache()

ClearCache removes all cached entries.

func (*CachedProvider) Name

func (cp *CachedProvider) Name() string

Name returns the inner provider's name.

func (*CachedProvider) Ping

func (cp *CachedProvider) Ping(ctx context.Context) error

Ping delegates to the inner provider (no caching).

func (*CachedProvider) SetEnabled

func (cp *CachedProvider) SetEnabled(enabled bool)

SetEnabled toggles caching at runtime.

func (*CachedProvider) StreamChat

func (cp *CachedProvider) StreamChat(ctx context.Context, messages []RelayMessage, opts ChatOptions) (*StreamResult, error)

StreamChat delegates to the inner provider without caching. Streaming responses are inherently incremental and not suitable for simple response caching.

type ChatOptions

type ChatOptions = core.ChatOptions

Cache decorators use the shared provider contract; cache has no dependency on the provider composition root.

type Provider

type Provider = core.Provider

Cache decorators use the shared provider contract; cache has no dependency on the provider composition root.

type RelayMessage

type RelayMessage = core.RelayMessage

Cache decorators use the shared provider contract; cache has no dependency on the provider composition root.

type RelayResponse

type RelayResponse = core.RelayResponse

Cache decorators use the shared provider contract; cache has no dependency on the provider composition root.

type StreamResult

type StreamResult = core.StreamResult

Cache decorators use the shared provider contract; cache has no dependency on the provider composition root.