sanecache

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 5, 2026 License: MIT Imports: 9 Imported by: 0

README

sanecache

CI Go Reference Go Report Card

A small in-memory cache for Go that aims to be predictable before it is fast.

There is no shortage of Go caches, and several of them are excellent. This one exists because the failures that actually cost time in production were never about throughput: a write that reports success and is dropped a moment later, a hit rate that quietly collapses because every value is too big for the budget, a limit expressed in entries when the thing you are protecting is a memory limit, and a batch of keys that all expire in the same millisecond and stampede the upstream together.

import "github.com/andared/sanecache"

Requires Go 1.24. No dependencies.

Quick start

c := sanecache.New(sanecache.Options[string, *Article]{
    TTL:         10 * time.Minute,
    NegativeTTL: 30 * time.Second,
    Jitter:      10,
    MaxBytes:    64 << 20,
    Cost:        func(a *Article) int64 { return a.ApproxBytes() },
})
defer c.Close()

if v, status := c.Lookup(id); status != sanecache.StatusMiss {
    if status == sanecache.StatusNegative {
        return nil, ErrNotFound // the upstream already told us, don't ask again
    }
    return v, nil
}

article, err := fetch(ctx, id)
switch {
case errors.Is(err, ErrNotFound):
    c.SetNegative(id)
    return nil, err
case err != nil:
    return nil, err
}

if err := c.Set(id, article); err != nil {
    // ErrTooLarge: this key will never be cached. Worth a metric.
}

What "sane" means here

Writes are synchronous. A value is readable the moment Set returns. Caches that buffer writes asynchronously are faster on paper and force Wait() calls into your tests to paper over the gap — and once your tests need it, so does any code that writes a value and reads it back in the same request.

A value that does not fit is refused, not swallowed. Set returns ErrTooLarge when the value's cost exceeds its shard's budget. The alternative — accept the write, return success, drop the entry during eviction — is invisible: the hit rate does not obviously look wrong, and every request for those keys goes to the upstream forever.

Budgets are in bytes. A cap on the number of entries tells you nothing about memory when entries are documents rather than integers: ten thousand entries is a rounding error for int values and several gigabytes for HTML templates. MaxBytes requires a Cost function; asking for a byte budget without one is a construction-time panic rather than a budget that silently counts entries.

"It does not exist" is an answer. SetNegative records that the upstream was asked and said no. Without a first-class form for this, it ends up as a sentinel value smuggled inside your value type, which does not survive the type parameter and tends to be charged zero cost — making it the one thing eviction can never reclaim.

TTLs can be jittered. Jitter: 10 spreads each expiry by up to ±10%. Keys warmed together by one request otherwise expire together and hit the upstream as one wave.

The Cost function is the part worth getting right

MaxBytes is only as honest as Cost. The number you want is resident size, and it is usually several times the serialized size — a decoded struct carries headers, pointers, map overhead and per-field padding that JSON does not.

Measure it once instead of guessing:

runtime.GC()
var before runtime.MemStats
runtime.ReadMemStats(&before)

values := make([]*Article, 0, n)
for range n {
    values = append(values, decode(sample))
}

runtime.GC()
var after runtime.MemStats
runtime.ReadMemStats(&after)
runtime.KeepAlive(values)

ratio := float64(after.HeapAlloc-before.HeapAlloc) / float64(n*len(sample))

In one production service this ratio came out at ~2.6× for decoded JSON structs, higher for map[string]any, and ~6.7× for compiled templates. Sizing a cache off raw payload length understated real memory sevenfold.

Sharding

Shards splits the cache into independently locked parts (rounded up to a power of two). It is off by default, because it is not free: each shard gets an equal slice of MaxBytes, so an uneven key distribution leaves part of the budget unused, and eviction becomes per-shard rather than global.

Whether it is worth it depends entirely on contention. On darwin/arm64 with GOMAXPROCS=8, 4096 keys, all readers hitting one cache:

shards Get (LRU) Get (ClearOnFull) Set 90/10 mixed
1 146 ns 89 ns 205 ns 146 ns
4 91 ns 68 ns 129 ns 85 ns
16 74 ns 60 ns 86 ns 62 ns
64 68 ns 70 ns 81 ns 95 ns

Two things to read out of that. Sharding buys roughly 2× on this machine and stops paying past ~16 shards — past that you are adding shards faster than you are removing contention. And the single-threaded cost of Get is 58 ns, of which 29 ns is time.Now() — half of a lookup is reading the clock to check expiry, not touching the map. If your service does one lookup per request, none of this matters and Shards: 0 is the right answer.

Eviction policies

LRU (default) evicts least-recently-used entries until the shard fits. Maintaining that order means every read takes a write lock.

ClearOnFull drops the whole shard except the entry that just overflowed it. Reads then need only a read lock — about 40% cheaper above. This is the right trade when access order is flat (everything in the working set is used every cycle, so LRU has no information to offer) and refilling is cheap relative to the bookkeeping.

Stats

Counters are built in — no callback interface on the hot path. Poll Stats() on your metrics interval and export it however you like:

s := c.Stats()
// s.Hits, s.Misses, s.Negatives, s.Evictions, s.Expirations,
// s.Replacements, s.Rejections, s.Entries, s.Bytes, s.HitRate()

Rejections is the one to alert on: a steady nonzero rate means keys that can never be cached. OnEvict is available separately when entries hold resources that need releasing.

Lifecycle

A background goroutine sweeps expired entries so they stop occupying budget before anyone looks them up. Close() stops it. Forgetting to call Close() does not leak the goroutine — dropping the cache stops it too — but calling it is still better than relying on when the collector gets around to it.

DisableCleanup: true runs without the goroutine; entries then expire only on lookup.

When to use something else

  • You want the best possible hit rate for a given memory budget, and admission policies and access-frequency estimation are worth the complexity → otter.
  • You are caching hundreds of megabytes and GC pressure from millions of live pointers is your actual problem → bigcache or freecache, which keep entries off-heap.
  • You just want a bounded LRU with TTL and nothing else → hashicorp/golang-lru's v2/expirable.

Status

v0.1. The API above is what exists and is tested; expect it to move before v1.

On the list, in rough order: a loader with single-flight so a cold key is fetched once rather than once per concurrent caller; typed views over a shared byte budget, for when one budget holds several value types; and a coarse clock to take that 29 ns off the read path.

Contributions are welcome — see CONTRIBUTING.md for what this library optimises for before proposing a change.

MIT licensed.

Documentation

Overview

Package sanecache is a small in-memory cache that aims to be predictable before it is fast.

Writes are synchronous, so a value is readable the moment Set returns. A value that cannot fit is refused with an error instead of being accepted and dropped later. Budgets are expressed in bytes, because a limit on the number of entries says nothing about the memory a process will use when entries are documents rather than integers. "The upstream says this key does not exist" is a first class answer rather than a marker smuggled inside the value type. And TTLs can carry jitter, so a batch of keys warmed by one request does not expire in lockstep and stampede the upstream.

Example 
package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

type article struct {
	ID   string
	Body string
}

func main() {
	c := sanecache.New(sanecache.Options[string, *article]{
		TTL:      10 * time.Minute,
		Jitter:   10,
		MaxBytes: 1 << 20,
		// Resident size, not serialized size: a decoded struct costs several
		// times its payload. Measure the ratio rather than guessing it.
		Cost: func(a *article) int64 { return int64(len(a.Body)) * 3 },
	})
	defer c.Close()

	if err := c.Set("a1", &article{ID: "a1", Body: "hello"}); err != nil {
		fmt.Println("set:", err)
		return
	}

	if a, ok := c.Get("a1"); ok {
		fmt.Println("cached:", a.Body)
	}

}

Output:
cached: hello

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrTooLarge is returned by Set when the value's cost exceeds the budget of
	// the shard it hashes to, so caching it could never succeed. It is reported
	// rather than swallowed: a steady trickle of ErrTooLarge means every request
	// for those keys goes to the upstream, which is invisible in the hit rate.
	ErrTooLarge = errors.New("sanecache: value cost exceeds the shard budget")

	// ErrNegativeDisabled is returned by SetNegative when Options.NegativeTTL was
	// not set. Negative entries without a TTL would pin a "does not exist" answer
	// for the lifetime of the process.
	ErrNegativeDisabled = errors.New("sanecache: negative caching is disabled (Options.NegativeTTL is unset)")
)

Functions

This section is empty.

Types

type Cache 

type Cache[K comparable, V any] struct {
	// contains filtered or unexported fields
}

Cache is a sharded, TTL-based cache. It is safe for concurrent use. The zero value is not usable; call New.

func New 

func New[K comparable, V any](o Options[K, V]) *Cache[K, V]

New builds a cache from o. It panics on options that cannot describe a working cache, such as a byte budget without a Cost function: those are programming mistakes, and failing at construction beats a cache that silently misbehaves.

func (*Cache[K, V]) Bytes 

func (c *Cache[K, V]) Bytes() int64

Bytes reports the summed cost of the entries held.

func (*Cache[K, V]) Clear 

func (c *Cache[K, V]) Clear()

Clear empties the cache. OnEvict is not called.

func (*Cache[K, V]) Close 

func (c *Cache[K, V]) Close()

Close stops the background sweeper. The cache stays usable afterwards; entries then expire only on lookup. Calling Close more than once is safe, and a cache that is simply dropped stops its sweeper too.

func (*Cache[K, V]) Delete 

func (c *Cache[K, V]) Delete(key K) bool

Delete removes key and reports whether it was present. OnEvict is not called.

func (*Cache[K, V]) Get 

func (c *Cache[K, V]) Get(key K) (V, bool)

Get returns the cached value. A cached "does not exist" answer reports false, same as a miss; use Lookup to tell the two apart.

func (*Cache[K, V]) Len 

func (c *Cache[K, V]) Len() int

Len reports how many entries are held, including expired ones not yet swept.

func (*Cache[K, V]) Lookup 

func (c *Cache[K, V]) Lookup(key K) (V, Status)

Lookup returns the cached value and how the cache answered.

Example

Telling "nobody asked yet" apart from "the upstream says it does not exist". 

package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

func main() {
	c := sanecache.New(sanecache.Options[string, string]{
		TTL:         time.Minute,
		NegativeTTL: 30 * time.Second,
	})
	defer c.Close()

	_, status := c.Lookup("gone")
	fmt.Println("before asking the upstream:", status)

	// The upstream answered "no such key". Remember that, or every render of a
	// template naming this id will ask again.
	if err := c.SetNegative("gone"); err != nil {
		fmt.Println("set negative:", err)
		return
	}

	_, status = c.Lookup("gone")
	fmt.Println("after:", status)

	// Get collapses the two, because both mean "you have no value".
	_, ok := c.Get("gone")
	fmt.Println("Get reports a hit:", ok)

}

Output:
before asking the upstream: miss
after: negative
Get reports a hit: false

func (*Cache[K, V]) Set 

func (c *Cache[K, V]) Set(key K, value V) error

Set caches value under key for the configured TTL.

Example

A value that cannot fit is refused rather than accepted and dropped later. 

package main

import (
	"errors"
	"fmt"

	"github.com/andared/sanecache"
)

func main() {
	c := sanecache.New(sanecache.Options[string, []byte]{
		MaxBytes: 1024,
		Cost:     func(b []byte) int64 { return int64(len(b)) },
	})
	defer c.Close()

	err := c.Set("huge", make([]byte, 4096))
	fmt.Println("rejected:", errors.Is(err, sanecache.ErrTooLarge))

	// The rejection is counted, so keys that can never be cached show up as
	// their own signal instead of an unexplained miss rate.
	_, ok := c.Get("huge")
	fmt.Println("cached:", ok, "rejections:", c.Stats().Rejections)

}

Output:
rejected: true
cached: false rejections: 1

func (*Cache[K, V]) SetNegative 

func (c *Cache[K, V]) SetNegative(key K) error

SetNegative records that the upstream reports no such key, for the configured NegativeTTL. Without it, a template that names a deleted object hits the upstream on every single render.

func (*Cache[K, V]) SetNegativeTTL 

func (c *Cache[K, V]) SetNegativeTTL(key K, ttl time.Duration) error

SetNegativeTTL is SetNegative with an explicit lifetime.

func (*Cache[K, V]) SetTTL 

func (c *Cache[K, V]) SetTTL(key K, value V, ttl time.Duration) error

SetTTL caches value under key for ttl, overriding Options.TTL. A ttl of zero means the entry never expires on its own.

func (*Cache[K, V]) Stats 

func (c *Cache[K, V]) Stats() Stats

Stats returns a snapshot of the counters. It walks every shard, so poll it on a metrics interval rather than per request.

Example 
package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

func main() {
	c := sanecache.New(sanecache.Options[int, string]{
		TTL:         time.Minute,
		NegativeTTL: time.Minute,
	})
	defer c.Close()

	_ = c.Set(1, "one")
	_ = c.SetNegative(2)

	c.Get(1)    // hit
	c.Lookup(2) // negative
	c.Get(3)    // miss

	s := c.Stats()
	// A cached negative counts towards the hit rate: it saved the same upstream
	// call a positive answer would have.
	fmt.Printf("hits=%d negatives=%d misses=%d rate=%.2f\n",
		s.Hits, s.Negatives, s.Misses, s.HitRate())

}

Output:
hits=1 negatives=1 misses=1 rate=0.67

type EvictReason 

type EvictReason uint8

EvictReason explains why an entry left the cache. Explicit Delete and Clear calls do not report a reason. 

const (
	ReasonEvicted  EvictReason = iota // dropped to stay inside the budget
	ReasonExpired                     // its TTL ran out
	ReasonReplaced                    // a later Set overwrote the key
)

The reasons an entry can leave the cache on its own.

func (EvictReason) String 

func (r EvictReason) String() string

type Options 

type Options[K comparable, V any] struct {
	// TTL is how long a value stays valid. Zero means entries never expire on
	// their own, which only makes sense for a cache bounded by MaxBytes or
	// MaxEntries, or one whose entries all get an explicit TTL via SetTTL.
	TTL time.Duration

	// NegativeTTL enables SetNegative and sets how long a cached "does not
	// exist" answer lives. It is usually much shorter than TTL: an object that
	// does not exist yet may appear at any moment.
	NegativeTTL time.Duration

	// Jitter spreads expiry times by up to this percentage in either direction,
	// so keys written together do not expire together. Must be 0..100.
	Jitter int

	// MaxBytes is the total budget in bytes, split evenly across shards. It
	// requires Cost. Zero means unbounded.
	MaxBytes int64

	// MaxEntries caps the number of entries, split evenly across shards. Zero
	// means unbounded. Prefer MaxBytes unless entries are uniform in size.
	MaxEntries int

	// Cost reports the memory a value occupies, in bytes. It is what MaxBytes is
	// measured against, so it should approximate resident size rather than
	// serialized size: a decoded struct commonly costs several times its JSON.
	// Measure it once rather than guessing (see the README).
	Cost func(V) int64

	// Shards splits the cache into independently locked parts, rounded up to a
	// power of two. Zero and one both mean a single lock. More shards reduce
	// contention but make the budget approximate: each shard gets an equal slice
	// of MaxBytes, and an uneven key distribution leaves some of it unused.
	//
	// The per-shard slice is rounded up, so the shards together are never
	// stricter than what was asked for. With small caps that rounding dominates:
	// MaxEntries of 2 across 16 shards is one entry per shard, or sixteen in
	// total. Keep the cap comfortably larger than the shard count.
	Shards int

	// Policy selects the eviction strategy. Defaults to LRU.
	Policy Policy

	// CleanupInterval is how often a background goroutine drops expired entries.
	// Zero picks an interval from the configured TTLs. Expired entries are also
	// dropped lazily on lookup, but until they are swept they still count
	// against the budget.
	CleanupInterval time.Duration

	// DisableCleanup runs the cache without a background goroutine. Expiry then
	// happens only on lookup and on eviction.
	DisableCleanup bool

	// OnEvict, if set, is called for every entry that leaves the cache without
	// being explicitly deleted. Negative entries are reported with the zero
	// value. It runs outside the shard lock, on the goroutine that caused the
	// removal, so it must not block.
	OnEvict func(key K, value V, reason EvictReason)

	// DisableStats skips the counters behind Stats.
	DisableStats bool
}

Options configures a cache. The zero value is a valid, unbounded, never expiring cache; every field below is optional.

type Policy 

type Policy uint8

Policy decides what happens when a shard runs over its budget. 

const (
	// LRU evicts the least recently used entries until the shard fits again.
	// Keeping that order costs a write lock on every read.
	LRU Policy = iota

	// ClearOnFull drops the whole shard except the entry that overflowed it.
	// Reads then need only a read lock, which is worth more than precise
	// eviction when access order is flat and refilling is cheap relative to
	// the bookkeeping.
	ClearOnFull
)

func (Policy) String 

func (p Policy) String() string

type Stats 

type Stats struct {
	Hits         int64 // lookups that returned a value
	Misses       int64 // lookups that found nothing
	Negatives    int64 // lookups that found a cached "does not exist"
	Evictions    int64 // entries dropped to stay inside the budget
	Expirations  int64 // entries dropped because their TTL ran out
	Replacements int64 // entries overwritten by a later Set
	Rejections   int64 // Set calls refused with ErrTooLarge

	Entries int   // entries currently held, expired-but-not-yet-swept included
	Bytes   int64 // sum of the costs of those entries
}

Stats is a snapshot of the cache counters. Counters are cumulative since the cache was created; Entries and Bytes are instantaneous.

func (Stats) HitRate 

func (s Stats) HitRate() float64

HitRate reports hits as a fraction of all lookups. A cached negative answer counts as a hit: it saved the same upstream call a positive one would have.

type Status 

type Status uint8

Status is the outcome of a lookup. 

const (
	StatusMiss     Status = iota // the cache knows nothing about this key
	StatusHit                    // a value was cached
	StatusNegative               // the upstream was asked and said the key does not exist
)

The outcomes a lookup can report.

func (Status) String 

func (s Status) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.