Documentation
¶
Overview ¶
Package sfcache provides a local, thread-safe, fixed-size cache for expensive lookups with single-flight deduplication.
Problem ¶
Services that repeatedly fetch the same external value (DNS, secrets, remote metadata, API responses) often suffer from duplicated work under concurrency. Without coordination, multiple goroutines can trigger identical slow lookups at the same time, increasing latency, cost, and upstream load.
sfcache solves this by combining TTL caching, bounded memory, and in-flight request coalescing for identical keys.
How It Works ¶
Create a cache with New, providing:
- a LookupFunc that performs the external lookup,
- a maximum entry count (`size`),
- a time-to-live (`ttl`) for successful values,
- optional behavior overrides such as WithTTLFunc (per-entry TTLs) and WithStaleIfError (serve the last known good value on refresh failure).
On Cache.Lookup:
- If a non-expired entry exists, the cached value is returned immediately.
- If the key is being resolved by another goroutine, duplicate callers wait and receive that same result (single-flight behavior).
- On miss or expiry, one lookup function call is executed and its result is stored in cache.
- If cache capacity is reached, eviction removes an expired entry first, or otherwise the oldest entry by expiration deadline.
Key Features ¶
- Fixed-size local cache with explicit capacity to avoid unbounded memory growth.
- Internal synchronization for safe concurrent access without external locks.
- Lock-friendly fast path: cache hits only acquire a read lock.
- Single-flight request collapsing for duplicate in-flight lookups.
- TTL-based freshness with automatic refresh on next miss after expiry, with optional per-entry TTLs via WithTTLFunc.
- Optional stale-if-error resilience via WithStaleIfError.
- Monotonic-clock expiration, immune to wall-clock adjustments (e.g. NTP).
- Explicit cache control via Cache.Remove, Cache.Reset, and Cache.PurgeExpired.
Semantics and Caveats ¶
- Only successful values are cached for the TTL (including legitimate nil values). Lookup errors are shared with all coalesced callers but never cached (no negative caching); a failed key leaves an already-expired entry behind until it is lazily evicted or overwritten, and at capacity this residue can displace a healthy entry.
- Whatever the lookup function returns is passed through as-is, including a non-nil value returned alongside a non-nil error.
- A ttl <= 0 disables value caching: every call triggers a new lookup, but concurrent callers for the same key are still coalesced. Two options qualify this: WithTTLFunc can still assign individual entries a positive TTL, and with WithStaleIfError the previous value is retained and can be served after a failed refresh. A caller that waited for an in-flight lookup accepts the latest completed outcome for the key, which under heavy churn may come from a later flight than the one it first awaited.
- The external lookup runs under the context of the caller that started it. If the lookup fails with that context's error while coalesced waiters are queued, the error is not shared: one of the waiters retries the lookup with its own context. A caller whose own context ends receives ErrLookupAborted; no external lookup is ever started with an already-ended context, while fresh cached values are served regardless of context state.
- If a waiting caller's context ends at the same instant the awaited lookup completes, either outcome may be observed: the caller can receive ErrLookupAborted even though a result just became available, or the result even though its context just ended (select nondeterminism).
- Expiration uses the monotonic clock, which on most platforms (e.g. Linux CLOCK_MONOTONIC) does not advance while the system is suspended: TTLs are effectively extended by the time spent in suspend. This matters on laptops and sleeping virtual machines, not on always-on servers.
- The lookup function must honor context cancellation and eventually return: a lookup that hangs forever pins its key (in-flight entries never expire and are never evicted) until Cache.Remove or Cache.Reset. It must not call Cache.Lookup for the same key of the same cache, which would self-deadlock. If it panics, the panic propagates to the caller that ran it and waiters retry.
- With WithStaleIfError, a failed refresh serves the last known good value (with a nil error) until its original expiration plus maxStale; the revived entry stays expired so every call still attempts a refresh, and the first success replaces it. The stale window takes precedence over the context-induced retry, so an upstream that hangs (every refresh dying by caller timeout) is still served stale. Error residue is never served stale, and callers cannot distinguish a stale value from a fresh one. Stale protection is best-effort: the value is lost to capacity eviction (expired entries are evicted first), Cache.PurgeExpired, Cache.Remove, Cache.Reset, and a panicking lookup function.
- Cache.Remove and Cache.Reset also invalidate in-flight lookups: the result of a removed flight is returned to the caller that performed it but not cached, and coalesced waiters retry with a fresh lookup.
- The capacity bound can be exceeded while more than `size` distinct keys are being resolved at once, as in-flight entries are never evicted; the excess is reclaimed as those lookups complete and new entries are stored. Expired entries are removed lazily (or via Cache.PurgeExpired) and are counted by Cache.Len.
- Keys must be hashable and equal to themselves: interface keys holding unhashable dynamic types cause Cache.Lookup to panic, and keys containing NaN never match themselves, leaking unevictable entries until Cache.Reset.
- Cached values are returned by reference: callers must treat returned values as read-only.
Why It Matters ¶
- Reduces repeated network, database, or compute cost for hot keys.
- Improves throughput in high-concurrency workloads by collapsing duplicate calls.
- Keeps memory usage predictable with bounded capacity.
Usage ¶
The value type V is inferred from the lookup function, so Lookup returns typed values with no assertions:
cache := sfcache.New(func(ctx context.Context, key string) (*Customer, error) {
return fetchCustomer(ctx, key)
}, 256, 5*time.Minute)
customer, err := cache.Lookup(ctx, "customer:123")
if err != nil {
return err
}
_ = customer // *Customer
Optional behaviors are enabled with options:
cache := sfcache.New(lookupFn, 256, 5*time.Minute,
sfcache.WithTTLFunc(func(key string, c *Customer) time.Duration {
return c.TTL // freshness is a property of the data
}),
sfcache.WithStaleIfError[string, *Customer](10*time.Minute),
)
Example applications in this repository include:
- github.com/tecnickcom/nurago/pkg/awssecretcache
- github.com/tecnickcom/nurago/pkg/dnscache
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrLookupAborted = errors.New("sfcache: lookup aborted by context")
ErrLookupAborted is returned by Cache.Lookup when the caller's context ends while waiting for an in-flight lookup, or when the context has already ended before an external lookup would start. It wraps the context error, so errors.Is with context.Canceled or context.DeadlineExceeded keeps working.
var ErrNilLookupFunc = errors.New("sfcache: the lookup function is nil")
ErrNilLookupFunc is returned by Cache.Lookup when the cache was constructed with a nil lookup function.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache[K comparable, V any] struct { // contains filtered or unexported fields }
Cache is a generic, size-bounded single-flight cache with TTL expiration.
func New ¶
func New[K comparable, V any](lookupFn LookupFunc[K, V], size int, ttl time.Duration, opts ...Option[K, V]) *Cache[K, V]
New constructs a single-flight cache with the specified lookup function, max entries, and time-to-live. If lookupFn is nil, a default function is used that always fails with ErrNilLookupFunc. Capacity defaults to 1 if size <= 0. A ttl <= 0 disables value caching while still coalescing duplicate in-flight requests for the same key (unless overridden per entry via WithTTLFunc). The default behavior can be customized with options such as WithTTLFunc and WithStaleIfError.
func (*Cache[K, V]) Len ¶
Len returns the current number of entries in the cache, including expired entries that have not been evicted yet and in-flight lookup placeholders.
func (*Cache[K, V]) Lookup ¶
Lookup retrieves the value for a key, performing single-flight deduplication for concurrent requests. Returns cached value if not expired; coalesces duplicate in-flight requests; evicts old/expired entries on capacity. Only successful values are cached for the TTL: errors are not cached (no negative caching), so every error triggers a fresh lookup on the next call. With WithStaleIfError, a failed refresh may instead return the last known good value with a nil error. If the entry is removed (e.g. via Cache.Remove or Cache.Reset) while a lookup is in flight, the result is returned to its callers but not cached.
Example ¶
package main
import (
"context"
"fmt"
"time"
"github.com/tecnickcom/nurago/pkg/sfcache"
)
func main() {
// example lookup function that returns the key as value:
// the cache value type V is inferred from its return type.
lookupFn := func(_ context.Context, key string) (string, error) {
return key, nil
}
// create a new cache with a lookupFn function, a maximum number of 3 entries, and a TTL of 1 minute.
c := sfcache.New(lookupFn, 3, 1*time.Minute)
val, err := c.Lookup(context.TODO(), "some_key")
fmt.Println(val, err)
}
Output: some_key <nil>
func (*Cache[K, V]) PurgeExpired ¶
PurgeExpired removes all expired completed entries from the cache and returns the number of entries removed. In-flight lookups are not affected. NOTE: revived stale values (see WithStaleIfError) are stored as expired entries and are therefore purged too, forfeiting stale protection for their keys until the next successful lookup.
Example ¶
package main
import (
"context"
"fmt"
"time"
"github.com/tecnickcom/nurago/pkg/sfcache"
)
func main() {
lookupFn := func(_ context.Context, key string) (string, error) {
return key, nil
}
c := sfcache.New(lookupFn, 8, 10*time.Millisecond)
_, _ = c.Lookup(context.TODO(), "some_key")
time.Sleep(20 * time.Millisecond) // let the entry expire
// Expired entries are otherwise only removed lazily.
fmt.Println(c.PurgeExpired(), c.Len())
}
Output: 1 0
type LookupFunc ¶
type LookupFunc[K comparable, V any] func(ctx context.Context, key K) (V, error)
LookupFunc is the generic function signature for external lookup calls.
type Option ¶
type Option[K comparable, V any] func(c *Cache[K, V])
Option is a type to allow setting custom cache options.
func WithStaleIfError ¶
func WithStaleIfError[K comparable, V any](maxStale time.Duration) Option[K, V]
WithStaleIfError enables serving the last known good value when a refresh fails: if a lookup for an expired key returns an error, the previous successful value is returned (with a nil error) instead, but only until its original expiration plus maxStale.
The revived entry stays expired, so every subsequent call still attempts a fresh lookup (coalesced as usual): recovery is automatic on the first success, which resets the entry and its expiration. The stale window takes precedence over the context-induced waiter retry, so an upstream that hangs (every refresh failing by caller timeout) is still served stale; outside the window, context-induced failures keep their retry semantics. Entries whose last outcome was an error are never served stale. Callers cannot distinguish a stale value from a fresh one, and a coalesced waiter may observe the stale value marginally past maxStale (scheduling delay). A maxStale <= 0 disables the behavior (default).
Stale protection is best-effort, not guaranteed retention: the retained value rides on expired entries and is therefore lost to capacity eviction (expired entries are evicted first), Cache.PurgeExpired, Cache.Remove, Cache.Reset, and a panicking lookup function.
NOTE: the type parameters cannot be inferred because maxStale does not mention them, so this option requires explicit instantiation matching the cache types, e.g.:
sfcache.New(lookupFn, 128, ttl, sfcache.WithStaleIfError[string, []string](time.Minute))
Example ¶
package main
import (
"context"
"errors"
"fmt"
"time"
"github.com/tecnickcom/nurago/pkg/sfcache"
)
func main() {
var calls int
// The first lookup succeeds, then the upstream becomes unavailable.
lookupFn := func(_ context.Context, _ string) (string, error) {
calls++
if calls == 1 {
return "value-1", nil
}
return "", errors.New("upstream outage")
}
// NOTE: WithStaleIfError requires explicit type instantiation matching
// the cache types, because its argument does not mention them.
c := sfcache.New(lookupFn, 8, 10*time.Millisecond,
sfcache.WithStaleIfError[string, string](1*time.Minute),
)
val, err := c.Lookup(context.TODO(), "some_key")
fmt.Println(val, err)
time.Sleep(20 * time.Millisecond) // let the entry expire
// The refresh fails: the last known good value is served instead.
val, err = c.Lookup(context.TODO(), "some_key")
fmt.Println(val, err)
}
Output: value-1 <nil> value-1 <nil>
func WithTTLFunc ¶
func WithTTLFunc[K comparable, V any](ttlFn TTLFunc[K, V]) Option[K, V]
WithTTLFunc overrides the cache-wide TTL for individual entries.
After each successful lookup, ttlFn is called with the key and the value about to be cached: a positive result becomes that entry's TTL, while a zero or negative result falls back to the cache-wide TTL passed to New. A nil ttlFn leaves the cache-wide TTL in effect for every entry. Values revived by WithStaleIfError bypass ttlFn: a stale serve is not a successful lookup.
NOTE: ttlFn runs synchronously while the cache's internal lock is held: it must be fast (a slow ttlFn stalls all cache traffic) and it must not call any method of the same cache, which would self-deadlock.
Use this when freshness is a property of the data itself, such as caching DNS records with their authoritative TTL or credentials with a known expiration.
Example ¶
package main
import (
"context"
"fmt"
"time"
"github.com/tecnickcom/nurago/pkg/sfcache"
)
func main() {
type record struct {
addr string
ttl time.Duration
}
// The looked-up data carries its own freshness.
lookupFn := func(_ context.Context, _ string) (record, error) {
return record{addr: "192.0.2.1", ttl: 30 * time.Second}, nil
}
// Each entry is cached for the TTL carried by its value instead of the
// cache-wide default.
c := sfcache.New(lookupFn, 8, 1*time.Minute,
sfcache.WithTTLFunc(func(_ string, r record) time.Duration {
return r.ttl
}),
)
r, err := c.Lookup(context.TODO(), "example.com")
fmt.Println(r.addr, err)
}
Output: 192.0.2.1 <nil>
Source Files