hookcore

Documentation

Overview

Package hookcore defines the storage abstraction behind the compression pipeline (internal/hook) so it can be driven by either the on-disk cache (CLI, one-shot per invocation) or an in-memory store (a future long-lived daemon). It depends on internal/cache but never on internal/hook, so hook can safely import hookcore without an import cycle.

Index

• type MemStore
  • func NewMemStore() *MemStore
  • func (m *MemStore) FlushDirty()
  • func (m *MemStore) LastGet(key string) (string, bool)
  • func (m *MemStore) LastPut(key, content string)
  • func (m *MemStore) LoadSession(id string) *cache.SessionState
  • func (m *MemStore) PruneUsage(maxAge time.Duration)
  • func (m *MemStore) RefGet(hash string) (string, bool)
  • func (m *MemStore) RefHit(hash string)
  • func (m *MemStore) RefPut(hash, content string)
  • func (m *MemStore) RefSeen(hash string) bool
  • func (m *MemStore) SaveSession(id string, s *cache.SessionState)
  • func (m *MemStore) StateStore() StateStore
• type StateStore
  • func NewDiskStore() StateStore

Types

type MemStore

type MemStore struct {
	// contains filtered or unexported fields
}

MemStore is a fully in-RAM implementation of StateStore for the qdf-hookd daemon: session state lives in sharded/mutex-guarded maps instead of hitting disk on every hook call. Sessions are the only content held in RAM and flushed lazily; ref/last-output blob *content* is never held in RAM — refs/last are hash/key seen-sets only, and every RefPut/LastPut writes its blob straight to disk (via internal/cache) so a daemon restart or crash never loses a blob body. Usage (dedup-hit / access recency, for eviction scoring) is tracked in RAM and flushed lazily to the usage sidecars.

Safe for concurrent use by multiple goroutines. LoadSession returns a copy of the stored session (made under the shard's lock, with its own Files map), never the live pointer, so callers are always free to mutate the returned state in place. SaveSession is the only way a stored session changes: it atomically swaps in the given pointer under the shard's write lock, so a stored session is never mutated in place after being stored — safe for FlushDirty to read/marshal it (whether inside or outside the shard lock; see FlushDirty's doc comment for the fast/slow path split). There is no enforced single-writer-per-session invariant: two concurrent LoadSession/SaveSession round trips for the *same* id race at the application level and the later SaveSession simply overwrites the earlier one (last-writer-wins); neither call panics or corrupts memory, but one caller's edits can be lost.

func NewMemStore

func NewMemStore() *MemStore

NewMemStore builds a MemStore and scans existing on-disk state (refs under cache.RefsDir, last-output blobs under cache.LastOutDir) into the in-RAM seen-sets. Loading is best-effort: missing directories fail open (nothing is loaded for that store), and construction never fails because of them.

Sessions are deliberately NOT preloaded: the whole point of the weak session cache is that cold sessions do not occupy RAM, so eagerly decoding every session file at startup would defeat it (and spike RSS proportional to the number of sessions ever seen). Instead sessions are loaded lazily from disk on the first LoadSession that needs one, and cached weakly.

func (*MemStore) FlushDirty

func (m *MemStore) FlushDirty()

FlushDirty persists every dirty session to disk, then flushes the in-RAM usage indices (ref/last dedup-hit and access stats) to their sidecars. Ref/last-output *blobs* are never dirty here — RefPut/LastPut already wrote them straight to disk — so there is nothing to flush for them beyond usage.

Each dirty session is flushed from the STRONG pointer held in dirtySessions, not by looking it back up in the (weak) shard map. That strong ref both guarantees the session is still alive to flush and pins the exact bytes that were saved. No shard lock is needed to read it: a saved *SessionState is never mutated in place (LoadSession hands callers a copy, so every mutation lands on a fresh pointer that a later SaveSession swaps in), so the only concurrent access to this pointer's Files map is other read-only reads — a concurrent LoadSession's copySession — and concurrent map reads are safe. Persistence has two paths:

• Fast path (the common case): the session has at most cache.MaxSessionFiles entries, so cache.Save's automatic eviction would be a no-op. qdf-encode the strong-ref *SessionState directly (read-only) into a pooled buffer, then write to disk. This skips the copy entirely.
• Slow path (rare: a session actually over the cap): cache.Save's Evict step deletes entries from the passed state's Files map. Mutating the shared pointer's map would race a concurrent LoadSession's read of it — a fatal "concurrent map read and map write" (this crash is what copySession was introduced to fix; see TestMemStore_FlushEvictConcurrentLoad). So this path copies the session before handing it to cache.Save.

A session whose encode OR disk write fails keeps its strong ref (it is re-added to dirtySessions unless a newer save already superseded it), so a transient failure never drops the only copy of unpersisted state to the GC. After a successful persist the strong ref is dropped, which is exactly what makes the now-clean session GC-reclaimable via its remaining weak pointer. Finally the shards are swept of dead (collected) weak pointers so the maps stay bounded regardless of how many sessions have come and gone.

The dirty session set is swapped out under dirtyMu and iterated afterwards without holding it, so writes that arrive concurrently with a flush land in a fresh dirty set rather than being lost or blocking on disk I/O. The usage maps are cloned under usageMu for the same reason: SaveUsage's JSON marshaling happens outside the lock so a concurrent RefHit/usage bump never blocks on disk I/O.

func (*MemStore) LastGet

func (m *MemStore) LastGet(key string) (string, bool)

LastGet reads the previous tool output stored under key from disk on demand. MemStore never holds last-output content in RAM.

func (*MemStore) LastPut

func (m *MemStore) LastPut(key, content string)

LastPut writes the current tool output to disk under key and adds key to the seen-set. Like RefPut, this goes straight to disk — no dirty content held in RAM.

func (*MemStore) LoadSession

func (m *MemStore) LoadSession(id string) *cache.SessionState

LoadSession returns a copy of the session state for id. The copy has its own Files map, populated under the shard lock, so the caller can freely mutate the returned state (including its Files map) without racing a concurrent FlushDirty or another LoadSession/SaveSession round trip on the same id.

The shard map holds only weak pointers. There are three outcomes:

• Live hit: the weak pointer still resolves (the session is dirty, or another caller holds it) — copy and return it.
• Collected / absent: the session was never cached, or it was clean and the GC reclaimed its weak-only reference. Disk is the source of truth, so reload it with cache.Load (which itself returns a fresh empty state when no file exists), re-cache it weakly, and return a copy. This is transparent to callers — a reclaimed clean session is indistinguishable from a resident one except for the disk read.

This is last-writer-wins, not single-writer: if two goroutines both load the same session, mutate their copies, and save, the second SaveSession wins and the first goroutine's edits are silently lost. That is a callers' concern (real callers currently only drive one hook invocation per session id at a time); LoadSession/SaveSession themselves never panic or corrupt state no matter how many goroutines race on the same id.

func (*MemStore) PruneUsage

func (m *MemStore) PruneUsage(maxAge time.Duration)

PruneUsage drops entries from the in-RAM usageRefs/usageLast indices whose LastUsed is older than maxAge — the same TTL floor cache.PruneDir applies to the on-disk usage sidecars (nowSec-used > ttl ⇒ drop; see evict.go). Without this, Bump only ever grows usageRefs/usageLast (nothing in RefHit or LastPut ever deletes from them), so FlushDirty's periodic SaveUsage would keep rewriting every entry back into the sidecar — silently resurrecting whatever PruneDir had just pruned on disk, and growing the RAM maps without bound over the life of a long-running daemon.

Call this from the same tick that invokes cache.SweepBlobs (the daemon's sweep ticker) so the RAM map's retention exactly tracks the sidecar's TTL policy: after a sweep+flush cycle, both sidecar and RAM agree on what survived. Safe to call with an empty or nil-backed index.

func (*MemStore) RefGet

func (m *MemStore) RefGet(hash string) (string, bool)

RefGet reads the content stored under hash from disk on demand. MemStore never holds ref content in RAM.

func (*MemStore) RefHit

func (m *MemStore) RefHit(hash string)

RefHit records a dedup hit against hash: bumps its in-RAM usage stat (hits + last-access time), flushed lazily to the usage sidecar by FlushDirty. Guarded by usageMu, independent of refsMu (which guards only the seen-set).

func (*MemStore) RefPut

func (m *MemStore) RefPut(hash, content string)

RefPut writes content to disk under hash (lazy qdf-encoded blob) and adds hash to the seen-set. Unlike sessions, ref blobs are never held dirty in RAM — they go straight to disk here, so a daemon crash between RefPut and the next FlushDirty never loses a blob body.

func (*MemStore) RefSeen

func (m *MemStore) RefSeen(hash string) bool

RefSeen reports whether content addressed by hash was already stored (a seen-set lookup — never touches disk).

func (*MemStore) SaveSession

func (m *MemStore) SaveSession(id string, s *cache.SessionState)

SaveSession stores s as the current state for id and marks it dirty. The shard map gets a weak pointer (so the session becomes GC-reclaimable once clean and otherwise unreferenced), while dirtySessions keeps a strong pointer to the same s until FlushDirty persists it — the strong ref is what prevents a not-yet-flushed session from being collected out from under the weak pointer (see the dirtySessions field comment).

func (*MemStore) StateStore

func (m *MemStore) StateStore() StateStore

StateStore returns m as the StateStore interface. MemStore implements the interface directly, so this is an identity conversion (no adapter needed).

type StateStore

type StateStore interface {
	// LoadSession returns the session state for id (an empty state if none
	// exists yet).
	LoadSession(id string) *cache.SessionState
	// SaveSession persists the session state for id.
	SaveSession(id string, s *cache.SessionState)

	// RefSeen reports whether content addressed by hash was already stored.
	RefSeen(hash string) bool
	// RefPut stores content under hash.
	RefPut(hash, content string)
	// RefGet returns the content stored under hash.
	RefGet(hash string) (string, bool)
	// RefHit records a dedup hit against a stored ref (usage bump for eviction).
	RefHit(hash string)

	// LastGet returns the previous tool output stored under key.
	LastGet(key string) (string, bool)
	// LastPut stores the current tool output under key.
	LastPut(key, content string)
}

StateStore is the storage surface the compression pipeline needs: session state (per-file read/write tracking) plus the two content-addressed caches (dedup refs and per-tool-call last-output, used for delta encoding).

func NewDiskStore

func NewDiskStore() StateStore

NewDiskStore returns the on-disk StateStore backed by internal/cache.