Documentation
¶
Overview ¶
Package ttlmap provides a concurrent generic map with per-entry TTL, backed by xsync.Map.
It targets working sets of thousands to hundreds of thousands of entries; reads and writes are both concurrency-safe. Entries may expire after a duration or never expire (see NoExpiration). Expiry is handled in two layers: read paths such as Get/Has/GetTTL/GetWithTTL/SetTTL/Update lazily evict expired entries, while a background janitor uses an expiry min-heap to clean up only entries that have actually expired (O(expired)), with an O(1) idle tick, instead of a per-second full-table scan.
You must create a map with New (memory-only) or Open (snapshot persistence) and call Close to stop background goroutines promptly. If Close is not called, they stop when the map is garbage collected (see Close); no unsaved snapshot is written on GC. Persistence is disabled by default and must be enabled explicitly; see PersistConfig.
After Set/GetOrSet/Update/MSet, treat the stored value as immutable: if V contains a slice or pointer (e.g. []byte), only the header is copied, and mutating the underlying array from the caller races with Get and snapshot encoding.
Example ¶
package main
import (
"fmt"
"time"
ttlmap "github.com/bagualing/ttlmap_xsync"
)
func main() {
m := ttlmap.New[string, []byte](30 * time.Second)
defer m.Close()
m.Set("session", []byte("payload"), 0) // 0 = map default TTL
m.Set("config", []byte("cfg"), ttlmap.NoExpiration)
v, ok := m.Get("session")
fmt.Println(ok, string(v))
v, ok = m.Get("config")
fmt.Println(ok, string(v))
}
Output: true payload true cfg
Example (Snapshot) ¶
Example_snapshot demonstrates saving and restoring a snapshot: live entries are encoded to a writer and then restored into a new map from a reader.
package main
import (
"bytes"
"fmt"
"time"
ttlmap "github.com/bagualing/ttlmap_xsync"
)
func main() {
var buf bytes.Buffer
m := ttlmap.New[string, int](time.Minute)
m.Set("a", 1, time.Hour)
if err := m.WriteSnapshot(&buf, ttlmap.JSONCodec[string, int]{}); err != nil {
panic(err)
}
m.Close()
restored, err := ttlmap.Load(&buf, ttlmap.JSONCodec[string, int]{}, time.Minute)
if err != nil {
panic(err)
}
defer restored.Close()
v, _ := restored.Get("a")
fmt.Println(v)
}
Output: 1
Index ¶
- Constants
- Variables
- type BinaryCodec
- type Clock
- type Codec
- type JSONCodec
- type Map
- func Load[K comparable, V any](r io.Reader, codec Codec[K, V], defaultTTL time.Duration, opts ...Option) (*Map[K, V], error)
- func New[K comparable, V any](defaultTTL time.Duration, opts ...Option) *Map[K, V]
- func Open[K comparable, V any](defaultTTL time.Duration, cfg PersistConfig[K, V], opts ...Option) (*Map[K, V], error)
- func (m *Map[K, V]) All() iter.Seq2[K, V]
- func (m *Map[K, V]) Clear()
- func (m *Map[K, V]) Close() error
- func (m *Map[K, V]) DefaultTTL() time.Duration
- func (m *Map[K, V]) Delete(key K)
- func (m *Map[K, V]) DeleteExpired() int
- func (m *Map[K, V]) EnablePersistence(cfg PersistConfig[K, V]) error
- func (m *Map[K, V]) Get(key K) (V, bool)
- func (m *Map[K, V]) GetOrSet(key K, value V, ttl time.Duration) (V, bool)
- func (m *Map[K, V]) GetTTL(key K) (time.Duration, bool)
- func (m *Map[K, V]) GetWithTTL(key K) (V, time.Duration, bool)
- func (m *Map[K, V]) Has(key K) bool
- func (m *Map[K, V]) LoadAndDelete(key K) (V, bool)
- func (m *Map[K, V]) MGet(keys ...K) map[K]V
- func (m *Map[K, V]) MSet(values map[K]V, ttl time.Duration)
- func (m *Map[K, V]) Peek(key K) (V, bool)
- func (m *Map[K, V]) PersistErr() error
- func (m *Map[K, V]) PersistenceEnabled() bool
- func (m *Map[K, V]) Range(f func(key K, value V) bool)
- func (m *Map[K, V]) Save() error
- func (m *Map[K, V]) SaveFile(path string, codec Codec[K, V]) error
- func (m *Map[K, V]) Set(key K, value V, ttl time.Duration)
- func (m *Map[K, V]) SetIfAbsent(key K, value V, ttl time.Duration) bool
- func (m *Map[K, V]) SetTTL(key K, ttl time.Duration) bool
- func (m *Map[K, V]) Size() int
- func (m *Map[K, V]) Update(key K, ttl time.Duration, fn func(V) V) (v V, ok bool)
- func (m *Map[K, V]) WriteSnapshot(w io.Writer, codec Codec[K, V]) error
- type Option
- type PersistConfig
- type StringBinaryCodec
- type StringBytesCodec
Examples ¶
Constants ¶
const DefaultCleanupInterval = time.Second
DefaultCleanupInterval is the cleanup period used when WithCleanupInterval is not specified.
const DefaultExpiration time.Duration = 0
DefaultExpiration, when used as the ttl argument to Set/SetTTL/GetOrSet/Update/MSet, means "use the map default TTL".
const DefaultPersistInterval = 5 * time.Second
DefaultPersistInterval is used when PersistConfig.Interval is 0.
const NoExpiration time.Duration = -1
NoExpiration, when used as the ttl argument to Set/SetTTL/GetOrSet/Update/MSet, means the entry never expires. Any negative duration is treated the same way. GetTTL and GetWithTTL return NoExpiration for such entries.
Variables ¶
var ( ErrPersistenceNotEnabled = errors.New("ttlmap: persistence not enabled") ErrPersistenceAlreadyEnabled = errors.New("ttlmap: persistence already enabled") ErrClosed = errors.New("ttlmap: map is closed") ErrCorruptSnapshot = errors.New("ttlmap: corrupt snapshot") ErrUnsupportedSnapshot = errors.New("ttlmap: unsupported snapshot version") ErrSnapshotTooLarge = errors.New("ttlmap: snapshot exceeds configured limits") ErrSnapshotSync = errors.New("ttlmap: snapshot published but directory sync failed") )
Sentinel errors for snapshot persistence.
Functions ¶
This section is empty.
Types ¶
type BinaryCodec ¶
type BinaryCodec[K comparable, V any] struct{}
BinaryCodec encodes and decodes K and V via encoding.BinaryMarshaler / BinaryUnmarshaler. It suits user-defined structs: you decide the binary layout, and the size and speed are determined by the type's implementation.
UnmarshalBinary typically uses a pointer receiver; K and V should be struct value types (e.g. Map[ID, Item]), not pointer types.
func (BinaryCodec[K, V]) MarshalKey ¶
func (BinaryCodec[K, V]) MarshalKey(k K) ([]byte, error)
func (BinaryCodec[K, V]) MarshalValue ¶
func (BinaryCodec[K, V]) MarshalValue(v V) ([]byte, error)
func (BinaryCodec[K, V]) UnmarshalKey ¶
func (BinaryCodec[K, V]) UnmarshalKey(b []byte) (K, error)
func (BinaryCodec[K, V]) UnmarshalValue ¶
func (BinaryCodec[K, V]) UnmarshalValue(b []byte) (V, error)
type Clock ¶
Clock provides the current time. Tests can inject a fake implementation via WithClock. Now must be concurrency-safe: the janitor and map operations call it from multiple goroutines concurrently, and it must not block.
type Codec ¶
type Codec[K comparable, V any] interface { MarshalKey(K) ([]byte, error) UnmarshalKey([]byte) (K, error) MarshalValue(V) ([]byte, error) UnmarshalValue([]byte) (V, error) }
Codec serializes keys and values for snapshot persistence.
Marshal is called while a snapshot is being written: it must be fast, must not block, and must not call this Map's persistence methods (Save/SaveFile/WriteSnapshot/Close). On the Save/Close path, Marshal holds the persistence mutex, and re-entry deadlocks on that lock; on the WriteSnapshot/SaveFile path it does not hold that lock, and re-entry is infinite recursion (re-entering SaveFile also self-deadlocks against the cross-process file lock it holds). The core constraint is unchanged: Marshal must never re-enter persistence. Writes (Set/Delete etc.) do not deadlock but can make the snapshot being produced inconsistent, so avoid them. Marshal's return value may be retained by the snapshot writer until Save returns; implementations must not reuse the returned buffer across calls. If Unmarshal needs to keep the result, it must copy it out of b.
type JSONCodec ¶
type JSONCodec[K comparable, V any] struct{}
JSONCodec encodes keys and values with encoding/json.
func (JSONCodec[K, V]) MarshalKey ¶
func (JSONCodec[K, V]) MarshalValue ¶
func (JSONCodec[K, V]) UnmarshalKey ¶
func (JSONCodec[K, V]) UnmarshalValue ¶
type Map ¶
type Map[K comparable, V any] struct { // contains filtered or unexported fields }
Map is a concurrent map in which each key-value pair may carry a TTL or never expire. It must not be copied after first use; the zero value is invalid — use New or Open.
func Load ¶
func Load[K comparable, V any](r io.Reader, codec Codec[K, V], defaultTTL time.Duration, opts ...Option) (*Map[K, V], error)
Load decodes a snapshot from r and returns a new Map. defaultTTL follows the same rules as New; 0 returns an error instead of panicking. Expired records are skipped; never-expire records are kept. It does not enable persistence; if the result needs to be written to disk, call EnablePersistence or use Open instead.
func New ¶
New creates a Map with default TTL defaultTTL. defaultTTL must be > 0 or NoExpiration (any negative duration is stored as NoExpiration); 0 is invalid and panics. When the ttl argument to Set/SetTTL/GetOrSet/Update/MSet is DefaultExpiration (0), this default is used; a negative ttl means the entry never expires.
func Open ¶
func Open[K comparable, V any](defaultTTL time.Duration, cfg PersistConfig[K, V], opts ...Option) (*Map[K, V], error)
Open creates a Map with snapshot persistence. defaultTTL follows the same rules as New (> 0 or NoExpiration); 0 returns an error instead of panicking. If Path exists, its unexpired records are loaded into the new map (never-expire entries included). A missing file is not an error; a corrupt or incompatible file is. On failure the created map is closed.
func (*Map[K, V]) All ¶
All returns an iterator over unexpired key-value pairs. The same caveats as Range apply.
func (*Map[K, V]) Clear ¶
func (m *Map[K, V]) Clear()
Clear deletes all entries. Clear is not atomic: racing with concurrent Set, a few newly written entries may remain.
Note: this deliberately does not clear the expiry heap. Clearing the heap here would race with a concurrent Set's pushHeap — a write completed between "clear the map" and "clear the heap" would have its heap index wrongly removed, leaving that entry unreclaimable by the janitor once it expires. Residual stale heap entries are safe: after popping, the janitor re-reads the map to check for absence, and rebuilds the whole heap when it grows too large.
func (*Map[K, V]) Close ¶
Close stops the background goroutines (the janitor and the periodic snapshotter) and waits for them to exit. If persistence is enabled, it then writes a final snapshot and returns the write error (also recorded in PersistErr). Close is idempotent and concurrency-safe.
Note: Close only guarantees that writes completed before it began make it into the final snapshot; writes concurrent with or after Close no longer have a goroutine to flush them. Therefore stop all writes before calling Close/Save; if you keep writing after Close, you must call Save again explicitly to persist the new data. After Close, expired entries are cleaned up only by lazy eviction or DeleteExpired.
func (*Map[K, V]) DefaultTTL ¶
DefaultTTL returns the map-level default TTL. A negative value passed to New is normalized to NoExpiration.
func (*Map[K, V]) Delete ¶
func (m *Map[K, V]) Delete(key K)
Delete removes key, whether or not the entry has expired.
func (*Map[K, V]) DeleteExpired ¶
DeleteExpired deletes all currently expired entries and returns the number deleted. Never-expire entries are not deleted.
func (*Map[K, V]) EnablePersistence ¶
func (m *Map[K, V]) EnablePersistence(cfg PersistConfig[K, V]) error
EnablePersistence turns on snapshot persistence for an existing map. It does not read Path: use Open (or Load) for restore. If the map already contains entries, it marks dirty so the first periodic/Save write persists them.
It should be called before concurrent use. After Close it returns ErrClosed.
func (*Map[K, V]) Get ¶
Get returns the unexpired value for key. Never-expire entries are always live. A live hit is lock-free; an expired entry is deleted (taking the bucket lock) and treated as absent.
func (*Map[K, V]) GetOrSet ¶
GetOrSet atomically returns the unexpired value for key; if key is missing or expired, it stores value and returns it. The returned loaded matches sync.Map.LoadOrStore: true means an existing value was hit, false means this call performed the write.
Note: value is evaluated before the call, so GetOrSet only guarantees the atomicity of the write, not that "the same value is computed only once". For true single-flight computation, deduplicate before calling.
Example ¶
package main
import (
"fmt"
"time"
ttlmap "github.com/bagualing/ttlmap_xsync"
)
func main() {
m := ttlmap.New[string, int](time.Minute)
defer m.Close()
// First write (loaded=false means this call performed the store).
v, loaded := m.GetOrSet("hits", 1, time.Minute)
fmt.Println(v, loaded)
// Hit the existing value (loaded=true).
v, loaded = m.GetOrSet("hits", 99, time.Minute)
fmt.Println(v, loaded)
}
Output: 1 false 1 true
func (*Map[K, V]) GetTTL ¶
GetTTL returns the remaining TTL of an unexpired entry. Never-expire entries return NoExpiration. Missing or expired keys return ok=false; expired entries are deleted.
func (*Map[K, V]) GetWithTTL ¶
GetWithTTL returns the unexpired value for key along with its remaining TTL. Never-expire entries return remaining = NoExpiration. Expired entries are deleted.
func (*Map[K, V]) Has ¶
Has reports whether key has an unexpired value (including never-expire). Expired entries are deleted as a side effect.
func (*Map[K, V]) LoadAndDelete ¶
LoadAndDelete removes key, returning its value if it previously existed and was unexpired.
func (*Map[K, V]) MGet ¶
func (m *Map[K, V]) MGet(keys ...K) map[K]V
MGet reads unexpired values for multiple keys. It is not a globally atomic snapshot and does not delete expired entries.
func (*Map[K, V]) MSet ¶
MSet writes multiple values with the same TTL (interpreted as in Set). It is a convenience wrapper (sharing one now/deadline computation and one dirty mark), but underneath it is still a per-key Store: it is not globally atomic.
func (*Map[K, V]) Peek ¶
Peek returns the unexpired value for key (including never-expire entries) but does not delete the entry on expiry. The read is lock-free.
func (*Map[K, V]) PersistErr ¶
PersistErr returns the most recent snapshot write error, if any.
func (*Map[K, V]) PersistenceEnabled ¶
PersistenceEnabled reports whether Open or EnablePersistence has succeeded.
func (*Map[K, V]) Range ¶
Range calls f for each unexpired entry, including never-expire entries. Expired entries are skipped and are not deleted. Iteration stops when f returns false. xsync.Range copies each bucket then invokes the callback without holding the bucket lock, so f may safely call other methods of this Map (concurrent Sets during the iteration may not be reflected in this pass).
func (*Map[K, V]) Save ¶
Save writes a snapshot to the configured Path even if the map is not dirty. If persistence was never enabled it returns ErrPersistenceNotEnabled.
func (*Map[K, V]) SaveFile ¶
SaveFile atomically writes a snapshot to path. Persistence need not be enabled. The parent directory must already exist. It fsyncs the file and the parent directory (unlike Save with PersistConfig.NoSync). If persistence is already configured, use Save.
func (*Map[K, V]) Set ¶
Set stores value under key. ttl semantics: 0 / DefaultExpiration uses the map default TTL; a negative ttl / NoExpiration means never expire; ttl > 0 expires ttl from now. Do not mutate value's underlying data after Set.
func (*Map[K, V]) SetIfAbsent ¶
SetIfAbsent stores a value only if key is absent or expired, and reports whether a write occurred.
func (*Map[K, V]) SetTTL ¶
SetTTL replaces the TTL of an existing unexpired entry. It returns false if key is absent or expired (an expired entry is deleted). ttl is interpreted the same way as in Set.
func (*Map[K, V]) Size ¶
Size returns the number of stored entries (including expired ones not yet cleaned up). For the unexpired count, call DeleteExpired first.
func (*Map[K, V]) Update ¶
Update atomically performs one read-modify-write on key's current value: if it exists and is unexpired, the value is replaced with the result of fn(old value), the TTL is refreshed, and the new value and true are returned; otherwise the expired entry is deleted and the zero value and false are returned. fn runs under xsync's bucket lock: it must be fast, must not block, and must not call any method of this Map (that would re-enter and deadlock). If fn panics, the lock is released and this write is abandoned (the panic propagates unchanged), without blocking subsequent operations.
func (*Map[K, V]) WriteSnapshot ¶
WriteSnapshot encodes the unexpired entries to w, including never-expire entries. Persistence need not be enabled. Expired entries are skipped. The stream is a complete snapshot, not an incremental merge.
type Option ¶
type Option func(*options)
Option configures a Map.
func WithCapacity ¶
WithCapacity hints the initial capacity, which is passed to xsync.WithPresize. It is only a lower bound on capacity: it does not limit the number of entries nor trigger capacity-based eviction. A value <= 0 keeps the default.
func WithCleanupInterval ¶
WithCleanupInterval sets the background janitor's scan period. A value <= 0 disables the janitor, leaving expired entries to be handled only by lazy eviction or DeleteExpired.
type PersistConfig ¶
type PersistConfig[K comparable, V any] struct { // Path is the snapshot file path. Its parent directory must already exist. Required. Path string // Interval is the interval between dirty-snapshot writes. 0 means DefaultPersistInterval // (5s); a negative value disables periodic writes, taking a snapshot only on Save and Close. Interval time.Duration // Codec serializes keys and values. Required. Codec Codec[K, V] // NoSync skips both the file fsync and the parent-directory fsync. Faster, but more likely // to lose data on host crash. NoSync bool // OnError is called asynchronously when a periodic snapshot write fails (errors from // Close/Save are returned by their return values). It must not block for long. May be nil. // The callback runs in a separate goroutine; re-entering the same Map (including Close) is // safe, but on concurrent failures it may be skipped. OnError func(error) }
PersistConfig enables snapshot persistence for a Map. Unless you use Open or EnablePersistence, persistence stays disabled.
type StringBinaryCodec ¶
type StringBinaryCodec[V any] struct{}
StringBinaryCodec uses raw string keys and BinaryMarshaler values. It corresponds to the most common Map[string, struct] shape.
func (StringBinaryCodec[V]) MarshalKey ¶
func (StringBinaryCodec[V]) MarshalKey(k string) ([]byte, error)
func (StringBinaryCodec[V]) MarshalValue ¶
func (StringBinaryCodec[V]) MarshalValue(v V) ([]byte, error)
func (StringBinaryCodec[V]) UnmarshalKey ¶
func (StringBinaryCodec[V]) UnmarshalKey(b []byte) (string, error)
func (StringBinaryCodec[V]) UnmarshalValue ¶
func (StringBinaryCodec[V]) UnmarshalValue(b []byte) (V, error)
type StringBytesCodec ¶
type StringBytesCodec struct{}
StringBytesCodec encodes a Map[string, []byte] as raw bytes with no extra framing. It copies values to avoid the snapshot aliasing the caller's buffer.
func (StringBytesCodec) MarshalKey ¶
func (StringBytesCodec) MarshalKey(k string) ([]byte, error)
func (StringBytesCodec) MarshalValue ¶
func (StringBytesCodec) MarshalValue(v []byte) ([]byte, error)
func (StringBytesCodec) UnmarshalKey ¶
func (StringBytesCodec) UnmarshalKey(b []byte) (string, error)
func (StringBytesCodec) UnmarshalValue ¶
func (StringBytesCodec) UnmarshalValue(b []byte) ([]byte, error)