sanecache

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 6, 2026 License: MIT Imports: 12 Imported by: 0

README

sanecache

CI Go Reference Go Report Card

A small in-memory cache for Go that aims to be predictable before it is fast.

There is no shortage of Go caches, and several of them are excellent. This one exists because the failures that actually cost time in production were never about throughput: a write that reports success and is dropped a moment later, a hit rate that quietly collapses because every value is too big for the budget, a limit expressed in entries when the thing you are protecting is a memory limit, a batch of keys that all expire in the same millisecond and stampede the upstream together, and a cold key that a hundred concurrent requests each fetch for themselves.

import "github.com/andared/sanecache"

Requires Go 1.24. No dependencies.

Quick start

c := sanecache.New(sanecache.Options[string, *Article]{
    TTL:         10 * time.Minute,
    NegativeTTL: 30 * time.Second,
    Jitter:      10,
    MaxBytes:    64 << 20,
    Cost:        func(a *Article) int64 { return a.ApproxBytes() },
})
defer c.Close()

if v, status := c.Lookup(id); status != sanecache.StatusMiss {
    if status == sanecache.StatusNegative {
        return nil, ErrNotFound // the upstream already told us, don't ask again
    }
    return v, nil
}

article, err := fetch(ctx, id)
switch {
case errors.Is(err, ErrNotFound):
    c.SetNegative(id)
    return nil, err
case err != nil:
    return nil, err
}

if err := c.Set(id, article); err != nil {
    // ErrTooLarge: this key will never be cached. Worth a metric.
}

What "sane" means here

Writes are synchronous. A value is readable the moment Set returns. Caches that buffer writes asynchronously are faster on paper and force Wait() calls into your tests to paper over the gap — and once your tests need it, so does any code that writes a value and reads it back in the same request.

A value that does not fit is refused, not swallowed. Set returns ErrTooLarge when the value's cost exceeds its shard's budget. The alternative — accept the write, return success, drop the entry during eviction — is invisible: the hit rate does not obviously look wrong, and every request for those keys goes to the upstream forever.

Budgets are in bytes. A cap on the number of entries tells you nothing about memory when entries are documents rather than integers: ten thousand entries is a rounding error for int values and several gigabytes for HTML templates. MaxBytes requires a Cost function; asking for a byte budget without one is a construction-time panic rather than a budget that silently counts entries.

"It does not exist" is an answer. SetNegative records that the upstream was asked and said no. Without a first-class form for this, it ends up as a sentinel value smuggled inside your value type, which does not survive the type parameter and tends to be charged zero cost — making it the one thing eviction can never reclaim.

TTLs can be jittered. Jitter: 10 spreads each expiry by up to ±10%. Keys warmed together by one request otherwise expire together and hit the upstream as one wave.

A cold key is fetched once. GetOrLoad runs the loader once per key however many callers arrive while it is running.

Loading a cold key once

The block in the quick start — look up, go to the upstream on a miss, remember the answer, remember the absence of one — is the same in every service, and it has a hole in it: on a cold key, every concurrent request runs it at the same time.

c := sanecache.New(sanecache.Options[string, *Article]{
    TTL:         10 * time.Minute,
    NegativeTTL: 30 * time.Second,
    Loader: func(ctx context.Context, id string) (*Article, error) {
        a, err := db.Article(ctx, id)
        if errors.Is(err, sql.ErrNoRows) {
            return nil, sanecache.ErrNotFound // remember the absence too
        }
        return a, err
    },
})

article, err := c.GetOrLoad(ctx, id)
switch {
case errors.Is(err, sanecache.ErrNotFound):
    return nil, err // from the loader, or from a negative entry it left behind
case err != nil:
    return nil, err
}

Three details worth knowing, because they are where implementations differ:

"Does not exist" is one error, wherever it came from. The loader returns ErrNotFound; GetOrLoad caches that as a negative entry and reports the same ErrNotFound to later callers. Handling of "no such object" does not depend on whether the cache happened to remember it. Translate the upstream's own error once, in the loader, and it stops mattering anywhere else.

Giving up does not cancel the load for everyone else. The loader's context is not the first caller's. It carries that caller's values, but it is cancelled only when every caller waiting on the result has gone. golang.org/x/sync/singleflight hands the load the first caller's context, so a request that times out takes down the load that the other ninety-nine are waiting on. A caller who gives up here gets ctx.Err() and leaves the others alone.

Failures are not cached. Anything other than ErrNotFound is passed back unchanged and nothing is stored, so the next call tries again. Single-flight already collapses the retry storm; caching the error on top of that would turn a blip into an outage that outlives it.

The loader runs on a goroutine of the cache's own, which is what makes the two points above possible. If it panics, the panic is carried to the callers rather than taking the process down, and it arrives with the stack of where it actually happened. A load that needs a deadline of its own should set one inside the loader, where the right number is known.

A load with no caller left is cancelled — but a loader that does not watch its context finishes anyway, and its value is cached even then. That is deliberate: when callers time out faster than the upstream answers, throwing those values away means the cache never warms and every request keeps timing out. The price is the narrow case where such a load lands after a later one and puts back a value read before it, with the TTL starting again. A loader that honours cancellation never reaches it.

Several value types under one budget

A budget in bytes is only worth having if it covers everything competing for the memory. One cache per type means one budget per type, and dividing a fixed amount of memory between types up front is exactly the guess the byte budget was meant to avoid: the split that was right at deploy time is wrong by the next traffic pattern.

c := sanecache.New(sanecache.Options[string, any]{
    TTL:      10 * time.Minute,
    MaxBytes: 64 << 20,
    Cost:     func(any) int64 { return 256 }, // fallback for views without their own
})
defer c.Close()

articles := sanecache.NewView(c, sanecache.ViewOptions[*Article]{
    Name: "article",
    Cost: func(a *Article) int64 { return a.ApproxBytes() },
})
seasons := sanecache.NewView(c, sanecache.ViewOptions[*Season]{
    Name: "season",
    TTL:  time.Hour, // seasons change less often than articles do
})

a, ok := articles.Get(id) // a is a *Article, not an any

A view fixes one value type, gives each of its own counters, and prefixes its keys with its name, so two views cannot collide on the same id. It can bring its own Cost — which is what spares you the type switch that a shared Cost func(any) int64 otherwise becomes — and its own TTLs. Eviction stays global: a view that suddenly needs more memory takes it from whichever entries were used least recently, whatever type they are.

A view is a function rather than a method on Cache because a method cannot introduce a type parameter of its own. Reads cost about 6 ns more than the cache underneath, and no allocation while the name and key together stay within 32 bytes: up to that length the compiler keeps the joined key on the stack, and past it every read allocates.

Stats().TypeMisses counts lookups that found some other type under a view's key. With names in the keys that should never happen, which is the point: it is a bug detector for two views sharing a name, or a write made straight to the underlying cache.

The Cost function is the part worth getting right

MaxBytes is only as honest as Cost. The number you want is resident size, and it is usually several times the serialized size — a decoded struct carries headers, pointers, map overhead and per-field padding that JSON does not.

Measure it once instead of guessing:

runtime.GC()
var before runtime.MemStats
runtime.ReadMemStats(&before)

values := make([]*Article, 0, n)
for range n {
    values = append(values, decode(sample))
}

runtime.GC()
var after runtime.MemStats
runtime.ReadMemStats(&after)
runtime.KeepAlive(values)

ratio := float64(after.HeapAlloc-before.HeapAlloc) / float64(n*len(sample))

In one production service this ratio came out at ~2.6× for decoded JSON structs, higher for map[string]any, and ~6.7× for compiled templates. Sizing a cache off raw payload length understated real memory sevenfold.

Sharding

Shards splits the cache into independently locked parts (rounded up to a power of two). It is off by default, because it is not free: each shard gets an equal slice of MaxBytes, so an uneven key distribution leaves part of the budget unused, and eviction becomes per-shard rather than global.

Whether it is worth it depends entirely on contention. On an M3 with GOMAXPROCS=8, 4096 keys, all readers hitting one cache:

shards Get (LRU) Get (ClearOnFull) Set 90/10 mixed
1 131 ns 108 ns 195 ns 145 ns
4 66 ns 37 ns 116 ns 76 ns
16 40 ns 24 ns 76 ns 47 ns
64 33 ns 19 ns 65 ns 39 ns

Sharding is worth about 4x here and is still buying something at 64. If your service does one lookup per request, none of this matters and Shards: 0 is the right answer.

Eviction policies

LRU (default) evicts least-recently-used entries until the shard fits. Maintaining that order means every read takes a write lock.

ClearOnFull drops the whole shard except the entry that just overflowed it. Reads then need only a read lock, which is worth 20% on a single shard and 40% once the cache is sharded — a read lock only pays off when the cores taking it are not all queued behind the same one. This is the right trade when access order is flat (everything in the working set is used every cycle, so LRU has no information to offer) and refilling is cheap relative to the bookkeeping. It costs hit rate, though: see the table further down.

The clock

Uncontended, a Get that hits costs 47 ns on the machine above, and 27 ns of that is reading the wall clock to decide whether the entry has expired. Well over half of a lookup is time.Now().

ClockGranularity hands that job to a background goroutine and lets lookups read an atomic instead:

sanecache.Options[string, *Article]{
    TTL:              10 * time.Minute,
    ClockGranularity: 100 * time.Millisecond,
}

A hit then costs 20 ns rather than 47, and expiry becomes accurate to within one interval in either direction — which against a ten-minute TTL is nothing, and against a one-second TTL is a lot. It is off by default because a TTL that quietly means something other than what it says is exactly the kind of surprise this library is about. Close stops the goroutine and lookups go back to the wall clock, rather than to a clock that has stopped.

Stats

Counters are built in — no callback interface on the hot path. Poll Stats() on your metrics interval and export it however you like:

s := c.Stats()
// s.Hits, s.Misses, s.Negatives, s.TypeMisses, s.Evictions, s.Expirations,
// s.Replacements, s.Rejections, s.Loads, s.LoadErrors, s.Coalesced,
// s.Entries, s.Bytes, s.HitRate()

Rejections is the one to alert on: a steady nonzero rate means keys that can never be cached. Coalesced against Loads says how much work the single-flight is actually saving — if they are equal, the cache is cold in a way worth looking at. OnEvict is available separately when entries hold resources that need releasing.

Lifecycle

A background goroutine sweeps expired entries so they stop occupying budget before anyone looks them up. Close() stops it, along with the clock goroutine if there is one. Forgetting to call Close() does not leak them — dropping the cache stops them too — but calling it is still better than relying on when the collector gets around to it.

DisableCleanup: true runs without the sweeper; entries then expire only on lookup.

How it compares

The benchmarks/ module measures this cache against the ones it would otherwise be replacing. It is a separate module so the root stays dependency-free; make bench-compare runs it. Same machine as above, 256-byte values, a budget sized to 4096 of them. The sanecache rows are its knobs, each one added to the row above it.

ns/op serial Get Get ×8 Set ×8 90/10 ×8
sanecache, defaults 61 139 226 151
⤷ Shards: 16 64 42 91 49
⤷ + ClockGranularity 34 32 84 42
⤷ + ClearOnFull 32 20 90 39
otter v2 77 14 263 30
ristretto v2 86 21 324 65
golang-lru v2/expirable 52 142 190 165
ttlcache v3 63 177 267 192

The first column is what one lookup costs; the second is what eight goroutines get out of the same machine. Read them together, because they say opposite things. Per operation this cache is cheaper than otter and ristretto — they spend their time maintaining a frequency sketch and a set of ring buffers that only pay off later. What they buy with it is scaling: otter turns eight cores into 5.4x the throughput, this one into 1.6x, and unsharded into less than 1x, because reads take a lock and locks are where cores queue.

So the honest shape of it is not "slower". It is: the same work per operation, and less of the machine used to do it in parallel. Whether that matters is a question about the read rate. At one lookup per request, the gap between 20 ns and 14 ns is six nanoseconds against a request budget measured in milliseconds. It starts to matter when one request does thousands of lookups, or when the cache more or less is the service.

Writes are the other way round: 91 ns against 190 to 324. An admission policy has to decide whether to accept each write and update its sketch, and that costs more than taking a lock does. A write-heavy cache is the case where this library is simply faster.

The number that usually matters more than any of those is how much of a fixed budget each policy turns into hits. 4096 entries against a key space of 100,000:

%hit zipf s=1.20 zipf s=1.01 zipf + scan
sanecache (LRU) 87.3 65.8 78.0
sanecache (ClearOnFull) 83.1 58.0 73.7
otter v2 88.9 71.4 80.8
ristretto v2 87.4 68.7 78.9
golang-lru expirable 87.3 65.8 78.0
ttlcache v3 87.3 65.8 78.0

When the hot set fits, every policy looks the same and the extra machinery buys 1.6 points. When it does not, W-TinyLFU is worth 5.6 points of hit rate over LRU — and 5.6 points of upstream traffic is worth more than every nanosecond in the table above it. That is the real reason to pick otter, and it is a better one than throughput.

The same table prices the cheap read lock: ClearOnFull buys its 40% by giving up 7.8 points of hit rate. It is a trade for caches whose access order is flat, not a free win.

When to use something else

  • You want the best possible hit rate for a given memory budget, and admission policies and access-frequency estimation are worth the complexity → otter. The hit-rate table above is the size of the prize, and it is the biggest number on this page.
  • Your read path is hot enough that how well a cache scales across cores is a real difference → otter or ristretto, and budget for Wait() in the tests.
  • You are caching hundreds of megabytes and GC pressure from millions of live pointers is your actual problem → bigcache or freecache, which keep entries off-heap.
  • You just want a bounded LRU with TTL and nothing else → hashicorp/golang-lru's v2/expirable.

Status

v0.2. The API above is what exists and is tested; expect it to move before v1.

Contributions are welcome — see CONTRIBUTING.md for what this library optimises for before proposing a change.

MIT licensed.

Documentation

Overview

Package sanecache is a small in-memory cache that aims to be predictable before it is fast.

Writes are synchronous, so a value is readable the moment Set returns. A value that cannot fit is refused with an error instead of being accepted and dropped later. Budgets are expressed in bytes, because a limit on the number of entries says nothing about the memory a process will use when entries are documents rather than integers. "The upstream says this key does not exist" is a first class answer rather than a marker smuggled inside the value type. TTLs can carry jitter, so a batch of keys warmed by one request does not expire in lockstep and stampede the upstream. And a cold key is fetched once rather than once per concurrent caller asking for it.

Example 
package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

type article struct {
	ID   string
	Body string
}

func main() {
	c := sanecache.New(sanecache.Options[string, *article]{
		TTL:      10 * time.Minute,
		Jitter:   10,
		MaxBytes: 1 << 20,
		// Resident size, not serialized size: a decoded struct costs several
		// times its payload. Measure the ratio rather than guessing it.
		Cost: func(a *article) int64 { return int64(len(a.Body)) * 3 },
	})
	defer c.Close()

	if err := c.Set("a1", &article{ID: "a1", Body: "hello"}); err != nil {
		fmt.Println("set:", err)
		return
	}

	if a, ok := c.Get("a1"); ok {
		fmt.Println("cached:", a.Body)
	}

}

Output:
cached: hello

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrTooLarge is returned by Set when the value's cost exceeds the budget of
	// the shard it hashes to, so caching it could never succeed. It is reported
	// rather than swallowed: a steady trickle of ErrTooLarge means every request
	// for those keys goes to the upstream, which is invisible in the hit rate.
	ErrTooLarge = errors.New("sanecache: value cost exceeds the shard budget")

	// ErrNegativeDisabled is returned by SetNegative when Options.NegativeTTL was
	// not set. Negative entries without a TTL would pin a "does not exist" answer
	// for the lifetime of the process.
	ErrNegativeDisabled = errors.New("sanecache: negative caching is disabled (Options.NegativeTTL is unset)")

	// ErrNotFound is how a loader says the upstream has no such key, and how
	// GetOrLoad reports that answer back — including when it comes from a cached
	// negative entry rather than a fresh call. It is the same error either way,
	// so a caller's handling of "no such object" does not depend on whether the
	// cache happened to remember it.
	//
	// A loader may wrap it: GetOrLoad tests with errors.Is and passes the
	// loader's own error through to the caller that triggered the load.
	ErrNotFound = errors.New("sanecache: the upstream has no such key")

	// ErrNoLoader is returned by GetOrLoad when Options.Loader was not set.
	ErrNoLoader = errors.New("sanecache: GetOrLoad requires Options.Loader")
)

Functions

This section is empty.

Types

type Cache 

type Cache[K comparable, V any] struct {
	// contains filtered or unexported fields
}

Cache is a sharded, TTL-based cache. It is safe for concurrent use. The zero value is not usable; call New.

func New 

func New[K comparable, V any](o Options[K, V]) *Cache[K, V]

New builds a cache from o. It panics on options that cannot describe a working cache, such as a byte budget without a Cost function: those are programming mistakes, and failing at construction beats a cache that silently misbehaves.

func (*Cache[K, V]) Bytes 

func (c *Cache[K, V]) Bytes() int64

Bytes reports the summed cost of the entries held.

func (*Cache[K, V]) Clear 

func (c *Cache[K, V]) Clear()

Clear empties the cache. OnEvict is not called.

func (*Cache[K, V]) Close 

func (c *Cache[K, V]) Close()

Close stops the background goroutines. The cache stays usable afterwards: entries then expire only on lookup, and a cache configured with ClockGranularity goes back to reading the wall clock. Calling Close more than once is safe, and a cache that is simply dropped stops its goroutines too.

func (*Cache[K, V]) Delete 

func (c *Cache[K, V]) Delete(key K) bool

Delete removes key and reports whether it was present. OnEvict is not called.

func (*Cache[K, V]) Get 

func (c *Cache[K, V]) Get(key K) (V, bool)

Get returns the cached value. A cached "does not exist" answer reports false, same as a miss; use Lookup to tell the two apart.

func (*Cache[K, V]) GetOrLoad added in v0.2.0 

func (c *Cache[K, V]) GetOrLoad(ctx context.Context, key K) (V, error)

GetOrLoad returns the cached value, calling Options.Loader when there is none. Callers that ask for the same key while a load is running wait for it instead of starting their own, so a cold key costs one upstream call rather than one per concurrent caller.

A cached "does not exist" answer is reported as ErrNotFound without calling the loader. A loaded value is stored before this returns, so the next caller finds it cached; a value too large for the budget is still returned, counted as a rejection rather than quietly retried forever.

Errors other than ErrNotFound are returned as the loader produced them and are not cached, so the next call tries again.

Example

A cold key is fetched once however many callers want it at the same moment. 

package main

import (
	"context"
	"errors"
	"fmt"
	"sync"
	"sync/atomic"
	"time"

	"github.com/andared/sanecache"
)

type article struct {
	ID   string
	Body string
}

func main() {
	var upstreamCalls atomic.Int64

	c := sanecache.New(sanecache.Options[string, *article]{
		TTL:         time.Minute,
		NegativeTTL: 10 * time.Second,
		Loader: func(_ context.Context, id string) (*article, error) {
			upstreamCalls.Add(1)
			if id == "gone" {
				// The upstream's own "no such row" is translated once, here, so
				// that callers see the same error whether the answer came from
				// the upstream or from the negative entry it left behind.
				return nil, sanecache.ErrNotFound
			}

			return &article{ID: id, Body: "hello"}, nil
		},
	})
	defer c.Close()

	var wg sync.WaitGroup
	for range 10 {
		wg.Add(1)
		go func() {
			defer wg.Done()
			if _, err := c.GetOrLoad(context.Background(), "a1"); err != nil {
				fmt.Println("load:", err)
			}
		}()
	}
	wg.Wait()

	_, err := c.GetOrLoad(context.Background(), "gone")
	fmt.Println("missing:", errors.Is(err, sanecache.ErrNotFound))

	// Ten callers, one upstream call — plus the one that came back empty.
	fmt.Println("upstream calls:", upstreamCalls.Load())

}

Output:
missing: true
upstream calls: 2

func (*Cache[K, V]) Len 

func (c *Cache[K, V]) Len() int

Len reports how many entries are held, including expired ones not yet swept.

func (*Cache[K, V]) Lookup 

func (c *Cache[K, V]) Lookup(key K) (V, Status)

Lookup returns the cached value and how the cache answered.

Example

Telling "nobody asked yet" apart from "the upstream says it does not exist". 

package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

func main() {
	c := sanecache.New(sanecache.Options[string, string]{
		TTL:         time.Minute,
		NegativeTTL: 30 * time.Second,
	})
	defer c.Close()

	_, status := c.Lookup("gone")
	fmt.Println("before asking the upstream:", status)

	// The upstream answered "no such key". Remember that, or every render of a
	// template naming this id will ask again.
	if err := c.SetNegative("gone"); err != nil {
		fmt.Println("set negative:", err)
		return
	}

	_, status = c.Lookup("gone")
	fmt.Println("after:", status)

	// Get collapses the two, because both mean "you have no value".
	_, ok := c.Get("gone")
	fmt.Println("Get reports a hit:", ok)

}

Output:
before asking the upstream: miss
after: negative
Get reports a hit: false

func (*Cache[K, V]) Set 

func (c *Cache[K, V]) Set(key K, value V) error

Set caches value under key for the configured TTL.

Example

A value that cannot fit is refused rather than accepted and dropped later. 

package main

import (
	"errors"
	"fmt"

	"github.com/andared/sanecache"
)

func main() {
	c := sanecache.New(sanecache.Options[string, []byte]{
		MaxBytes: 1024,
		Cost:     func(b []byte) int64 { return int64(len(b)) },
	})
	defer c.Close()

	err := c.Set("huge", make([]byte, 4096))
	fmt.Println("rejected:", errors.Is(err, sanecache.ErrTooLarge))

	// The rejection is counted, so keys that can never be cached show up as
	// their own signal instead of an unexplained miss rate.
	_, ok := c.Get("huge")
	fmt.Println("cached:", ok, "rejections:", c.Stats().Rejections)

}

Output:
rejected: true
cached: false rejections: 1

func (*Cache[K, V]) SetNegative 

func (c *Cache[K, V]) SetNegative(key K) error

SetNegative records that the upstream reports no such key, for the configured NegativeTTL. Without it, a template that names a deleted object hits the upstream on every single render.

func (*Cache[K, V]) SetNegativeTTL 

func (c *Cache[K, V]) SetNegativeTTL(key K, ttl time.Duration) error

SetNegativeTTL is SetNegative with an explicit lifetime.

func (*Cache[K, V]) SetTTL 

func (c *Cache[K, V]) SetTTL(key K, value V, ttl time.Duration) error

SetTTL caches value under key for ttl, overriding Options.TTL. A ttl of zero means the entry never expires on its own.

func (*Cache[K, V]) Stats 

func (c *Cache[K, V]) Stats() Stats

Stats returns a snapshot of the counters. It walks every shard, so poll it on a metrics interval rather than per request.

Example 
package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

func main() {
	c := sanecache.New(sanecache.Options[int, string]{
		TTL:         time.Minute,
		NegativeTTL: time.Minute,
	})
	defer c.Close()

	_ = c.Set(1, "one")
	_ = c.SetNegative(2)

	c.Get(1)    // hit
	c.Lookup(2) // negative
	c.Get(3)    // miss

	s := c.Stats()
	// A cached negative counts towards the hit rate: it saved the same upstream
	// call a positive answer would have.
	fmt.Printf("hits=%d negatives=%d misses=%d rate=%.2f\n",
		s.Hits, s.Negatives, s.Misses, s.HitRate())

}

Output:
hits=1 negatives=1 misses=1 rate=0.67

type EvictReason 

type EvictReason uint8

EvictReason explains why an entry left the cache. Explicit Delete and Clear calls do not report a reason. 

const (
	ReasonEvicted  EvictReason = iota // dropped to stay inside the budget
	ReasonExpired                     // its TTL ran out
	ReasonReplaced                    // a later Set overwrote the key
)

The reasons an entry can leave the cache on its own.

func (EvictReason) String 

func (r EvictReason) String() string

type Options 

type Options[K comparable, V any] struct {
	// TTL is how long a value stays valid. Zero means entries never expire on
	// their own, which only makes sense for a cache bounded by MaxBytes or
	// MaxEntries, or one whose entries all get an explicit TTL via SetTTL.
	TTL time.Duration

	// NegativeTTL enables SetNegative and sets how long a cached "does not
	// exist" answer lives. It is usually much shorter than TTL: an object that
	// does not exist yet may appear at any moment.
	NegativeTTL time.Duration

	// Jitter spreads expiry times by up to this percentage in either direction,
	// so keys written together do not expire together. Must be 0..100.
	Jitter int

	// MaxBytes is the total budget in bytes, split evenly across shards. It
	// requires Cost. Zero means unbounded.
	MaxBytes int64

	// MaxEntries caps the number of entries, split evenly across shards. Zero
	// means unbounded. Prefer MaxBytes unless entries are uniform in size.
	MaxEntries int

	// Cost reports the memory a value occupies, in bytes. It is what MaxBytes is
	// measured against, so it should approximate resident size rather than
	// serialized size: a decoded struct commonly costs several times its JSON.
	// Measure it once rather than guessing (see the README).
	Cost func(V) int64

	// Loader fetches a value that is not cached. It is what GetOrLoad calls, and
	// callers that ask for the same key while it is running share the one call
	// instead of each starting their own.
	//
	// Returning an error that wraps ErrNotFound means the upstream has no such
	// key: the answer is cached as a negative entry when NegativeTTL is set, and
	// reported to every waiting caller. Any other error is passed through
	// unchanged and is not cached.
	//
	// The context is not any one caller's: it carries the values of the caller
	// that started the load, but it is cancelled only once every caller waiting
	// for the result has given up. A loader must not call GetOrLoad on the same
	// cache and key, which would wait for itself.
	//
	// A load that nobody is waiting for any more is cancelled, but a loader that
	// does not watch its context finishes regardless, and its value is cached
	// even so. That is what keeps a cache warming when callers time out faster
	// than the upstream answers; the price is that such a load can land after a
	// later one and put back a value read before it, with the TTL starting over.
	// A loader that honours cancellation never gets there.
	Loader func(ctx context.Context, key K) (V, error)

	// Shards splits the cache into independently locked parts, rounded up to a
	// power of two. Zero and one both mean a single lock. More shards reduce
	// contention but make the budget approximate: each shard gets an equal slice
	// of MaxBytes, and an uneven key distribution leaves some of it unused.
	//
	// The per-shard slice is rounded up, so the shards together are never
	// stricter than what was asked for. With small caps that rounding dominates:
	// MaxEntries of 2 across 16 shards is one entry per shard, or sixteen in
	// total. Keep the cap comfortably larger than the shard count.
	Shards int

	// Policy selects the eviction strategy. Defaults to LRU.
	Policy Policy

	// CleanupInterval is how often a background goroutine drops expired entries.
	// Zero picks an interval from the configured TTLs. Expired entries are also
	// dropped lazily on lookup, but until they are swept they still count
	// against the budget.
	CleanupInterval time.Duration

	// DisableCleanup runs the cache without a background goroutine. Expiry then
	// happens only on lookup and on eviction.
	DisableCleanup bool

	// ClockGranularity trades TTL precision for lookup speed. Reading the wall
	// clock is about half the cost of a lookup, so a cache under enough load for
	// that to show up can have a background goroutine hold the time instead,
	// refreshed this often. Expiry is then accurate to within one interval in
	// either direction. Zero, the default, reads the clock on every operation.
	//
	// The goroutine is separate from the sweeper, so this works with
	// DisableCleanup. Close stops it, and lookups go back to the wall clock.
	ClockGranularity time.Duration

	// OnEvict, if set, is called for every entry that leaves the cache without
	// being explicitly deleted. Negative entries are reported with the zero
	// value. It runs outside the shard lock, on the goroutine that caused the
	// removal, so it must not block.
	OnEvict func(key K, value V, reason EvictReason)

	// DisableStats skips the counters behind Stats.
	DisableStats bool
}

Options configures a cache. The zero value is a valid, unbounded, never expiring cache; every field below is optional.

type Policy 

type Policy uint8

Policy decides what happens when a shard runs over its budget. 

const (
	// LRU evicts the least recently used entries until the shard fits again.
	// Keeping that order costs a write lock on every read.
	LRU Policy = iota

	// ClearOnFull drops the whole shard except the entry that overflowed it.
	// Reads then need only a read lock, which is worth more than precise
	// eviction when access order is flat and refilling is cheap relative to
	// the bookkeeping.
	ClearOnFull
)

func (Policy) String 

func (p Policy) String() string

type Stats 

type Stats struct {
	Hits         int64 // lookups that returned a value
	Misses       int64 // lookups that found nothing
	Negatives    int64 // lookups that found a cached "does not exist"
	Evictions    int64 // entries dropped to stay inside the budget
	Expirations  int64 // entries dropped because their TTL ran out
	Replacements int64 // entries overwritten by a later Set
	Rejections   int64 // Set calls refused with ErrTooLarge

	// TypeMisses counts view lookups that found an entry holding some other
	// type. Hits counts those too, because the cache did have the key; the pair
	// is what tells a namespace collision apart from a plain miss.
	TypeMisses int64

	Loads      int64 // Loader calls that finished, successfully or not
	LoadErrors int64 // of those, the ones that returned an error
	// Coalesced counts the GetOrLoad calls that another caller's load spared
	// from starting one of their own, whether they waited for it or arrived just
	// after it published. Against Loads it says how much the single flight is
	// actually saving.
	Coalesced int64

	Entries int   // entries currently held, expired-but-not-yet-swept included
	Bytes   int64 // sum of the costs of those entries
}

Stats is a snapshot of the cache counters. Counters are cumulative since the cache was created; Entries and Bytes are instantaneous.

func (Stats) HitRate 

func (s Stats) HitRate() float64

HitRate reports hits as a fraction of all lookups. A cached negative answer counts as a hit: it saved the same upstream call a positive one would have.

type Status 

type Status uint8

Status is the outcome of a lookup. 

const (
	StatusMiss     Status = iota // the cache knows nothing about this key
	StatusHit                    // a value was cached
	StatusNegative               // the upstream was asked and said the key does not exist
)

The outcomes a lookup can report.

func (Status) String 

func (s Status) String() string

type View added in v0.2.0 

type View[T any] struct {
	// contains filtered or unexported fields
}

View is a typed window onto a cache that holds values of several types under one byte budget. The cache is declared as Cache[string, any]; each view fixes one value type, namespaces its keys, and counts its own hits.

This is the shape that a budget in bytes forces. One cache per type would mean one budget per type, and splitting a fixed amount of memory between types up front is exactly the guess the byte budget was meant to avoid: the split that was right at deploy time is wrong by the next traffic pattern. A view is a function rather than a method on Cache because a method cannot introduce a type parameter of its own.

A view costs one string join per operation on top of the cache underneath. While the name and key together fit in 32 bytes the compiler keeps that on the stack; past it, every read allocates.

func NewView added in v0.2.0 

func NewView[T any](c *Cache[string, any], o ViewOptions[T]) *View[T]

NewView opens a view named o.Name onto c. It panics on a name that cannot keep views apart, for the same reason New panics on a budget it cannot honour.

Example

Several value types under one byte budget, which is the only kind of budget that does not need dividing up in advance. 

package main

import (
	"fmt"
	"time"

	"github.com/andared/sanecache"
)

type article struct {
	ID   string
	Body string
}

type season struct {
	Num int
}

func main() {
	c := sanecache.New(sanecache.Options[string, any]{
		TTL:      10 * time.Minute,
		MaxBytes: 64 << 20,
		// The fallback for views that do not bring their own Cost. With several
		// types sharing a budget, this is where the type switch would go.
		Cost: func(any) int64 { return 256 },
	})
	defer c.Close()

	articles := sanecache.NewView(c, sanecache.ViewOptions[*article]{
		Name: "article",
		Cost: func(a *article) int64 { return int64(len(a.Body)) * 3 },
	})
	seasons := sanecache.NewView(c, sanecache.ViewOptions[*season]{
		Name: "season",
		TTL:  time.Hour, // seasons change less often than articles do
	})

	_ = articles.Set("1", &article{ID: "1", Body: "hello"})
	_ = seasons.Set("1", &season{Num: 7})

	a, _ := articles.Get("1")
	s, _ := seasons.Get("1")
	fmt.Println(a.Body, s.Num)

	// The same key in two views is two entries: the view name is part of it.
	fmt.Println("entries:", c.Len(), "bytes:", c.Bytes())

}

Output:
hello 7
entries: 2 bytes: 271

func (*View[T]) Delete added in v0.2.0 

func (v *View[T]) Delete(key string) bool

Delete removes key from this view and reports whether it was present.

func (*View[T]) Get added in v0.2.0 

func (v *View[T]) Get(key string) (T, bool)

Get returns the cached value. As with Cache.Get, a cached "does not exist" answer reports false; use Lookup to tell it from a miss.

func (*View[T]) Lookup added in v0.2.0 

func (v *View[T]) Lookup(key string) (T, Status)

Lookup returns the cached value and how the view answered. An entry holding some other type is reported as a miss and counted as a TypeMiss: the value is unusable here, so the caller has to go to the upstream either way.

func (*View[T]) Name added in v0.2.0 

func (v *View[T]) Name() string

Name returns the view's name, which is also the prefix its keys carry in the underlying cache.

func (*View[T]) Set added in v0.2.0 

func (v *View[T]) Set(key string, value T) error

Set caches value under key for the view's TTL.

func (*View[T]) SetNegative added in v0.2.0 

func (v *View[T]) SetNegative(key string) error

SetNegative records that the upstream reports no such key, for the view's NegativeTTL.

func (*View[T]) SetNegativeTTL added in v0.2.0 

func (v *View[T]) SetNegativeTTL(key string, ttl time.Duration) error

SetNegativeTTL is SetNegative with an explicit lifetime.

func (*View[T]) SetTTL added in v0.2.0 

func (v *View[T]) SetTTL(key string, value T, ttl time.Duration) error

SetTTL caches value under key for ttl, overriding both the view's and the cache's TTL. A ttl of zero means the entry never expires on its own.

func (*View[T]) Stats added in v0.2.0 

func (v *View[T]) Stats() ViewStats

Stats returns a snapshot of this view's counters. The cache's own Stats counts the same lookups across every view.

type ViewOptions added in v0.2.0 

type ViewOptions[T any] struct {
	// Name identifies the view and namespaces its keys: the view stores under
	// Name + ":" + key. Two views therefore cannot collide, and an OnEvict
	// handler on the cache can tell whose entry it is looking at. It must not be
	// empty and must not contain ":".
	Name string

	// Cost reports the memory a value of this view occupies, in bytes, the way
	// Options.Cost does for the cache as a whole. A view that sets it is spared
	// the type switch that a shared Cost func(any) int64 turns into once a
	// budget holds several types. Unset, the cache's own Cost is used.
	Cost func(T) int64

	// TTL is how long this view's values stay valid. Zero takes the cache's TTL;
	// SetTTL still overrides both.
	//
	// A view with a much shorter TTL than the cache is worth a word of warning:
	// the sweeper's interval is chosen from the cache's TTLs, so those entries
	// may sit on the budget after expiring until a lookup or a sweep finds them.
	// Set Options.CleanupInterval when that matters.
	TTL time.Duration

	// NegativeTTL is how long this view's "does not exist" answers live. Zero
	// takes the cache's NegativeTTL, and if that is unset too, SetNegative on
	// this view reports ErrNegativeDisabled.
	NegativeTTL time.Duration
}

ViewOptions configures a view. Only Name is required.

type ViewStats added in v0.2.0 

type ViewStats struct {
	Hits      int64 // lookups that returned a value of this view's type
	Misses    int64 // lookups that found nothing
	Negatives int64 // lookups that found a cached "does not exist"

	// TypeMisses counts lookups that found an entry holding another type. It is
	// a bug detector rather than a routine metric: with keys namespaced by view
	// name, the only ways to get one are two views sharing a name and writes
	// made straight to the underlying cache.
	TypeMisses int64
}

ViewStats is a snapshot of one view's counters, cumulative since the view was opened.

func (ViewStats) HitRate added in v0.2.0 

func (s ViewStats) HitRate() float64

HitRate reports hits as a fraction of all lookups, counting a cached negative as a hit and a type miss as a miss.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.