README
¶
semcache — a correctness-aware semantic cache for LLM responses
Most semantic caches optimize one number: hit rate. In a regulated workload — banking, healthcare, anything audited — that framing is dangerous, because a wrong cache hit is a defect, not a missed optimization. Serving a confidently-cached answer to a question that was almost the same one is worse than calling the model.
semcache is built for that setting. It treats a bad hit as a bug and makes correctness a first-class, measurable property:
- Similarity floor + second-stage verification. A high cosine score is necessary but not sufficient. Every floor-clearing candidate goes through a pluggable verifier (lexical-overlap by default, or your own LLM-judge) before it is served.
- PII-aware keying. Raw PII never enters the index. Query text is redacted (synthetic Vietnamese shapes: CCCD, mobile, card, account, email) and normalized before it is embedded, keyed, or stored — so two queries that differ only by an account number correctly collapse to one entry, and secrets are never persisted.
- Deterministic staleness. Per-entry TTL plus an optional knowledge epoch: when a fact changes, bump the epoch and every answer that depended on it deterministically expires — no guessing.
- Negative cache. A verified near-miss is remembered as a negative anchor, so a semantically-close-but-different query is short-circuited to a miss instead of risking the wrong answer again.
- It measures the thing that matters. Stats report not just hit rate but
FalseHitsAvoided— candidates that cleared the floor and were rejected by verification. A generic cache never measures this.
The default path is zero-dependency and in-memory. Bring your own embedder; there is no provider lock-in and no implicit network call.
Install
go get github.com/shironeko2707/semcache
Requires Go 1.26+.
Quickstart
c, _ := semcache.New(semcache.NewHashEmbedder(256)) // swap in a real Embedder for production semantics
q := semcache.Query{Text: "what is the daily transfer limit?", Namespace: "faq"}
if entry, found, _ := c.Lookup(ctx, q); found {
return entry.Response
}
resp := callModel(ctx, q.Text)
_ = c.Store(ctx, q, semcache.Entry{Response: resp})
See examples/gateway_middleware.go for the LLM-gateway wrapper pattern (no import cycle, cache stays a library).
The design in one diagram
Lookup(query)
│
├─ redact PII + normalize ← raw PII never embedded/stored
├─ embed (your Embedder)
├─ Nearest(namespace, k) ← pluggable Store (in-memory default)
│
└─ for each candidate, best-first:
score < floor? → stop, miss
epoch mismatch? → skip (deterministic staleness)
meta mismatch? → skip (tenant/filter isolation)
negative anchor? → miss (known near-miss)
verify(query,cand)? → no → false-hit avoided, record negative, skip
yes → HIT
Bring your own embedder
HashEmbedder is a deterministic, dependency-free feature-hashing embedder — ideal for tests, benchmarks, and offline/air-gapped development. It is not a semantic model. For production semantics, implement the one-method interface around any model:
type Embedder interface {
Embed(ctx context.Context, text string) ([]float32, error)
}
Verification strategies
| Verifier | When |
|---|---|
LexicalVerifier{MinOverlap} (default) |
Cheap, no extra calls. Catches entity/amount swaps. Raise MinOverlap to be stricter. |
VerifierFunc (LLM-judge) |
Highest fidelity for cases lexical overlap misses (e.g. single-token negation). Off by default; opt in per deployment. |
FloorVerifier |
Floor-only. Only when false hits are genuinely cheap. |
Honest limitation: a single negation token ("... allowed" vs "... not allowed") yields high lexical overlap. The lexical verifier alone will not catch it at a low threshold — raise MinOverlap or add an LLM-judge verifier for that class of query. The library makes this a deliberate, configurable choice rather than a silent one.
Benchmark / exit criteria
make bench
On the synthetic repetitive banking workload (bench/), the reproducible targets are:
- Hit rate > 40% — current run: ~68% over a 60/40 repeat/long-tail mix.
- Near-miss probes wrongly served: 0 — the correctness headline.
- Lookup p99 < 5ms in-memory — current run: ~1ms.
All workload data is synthetic. Numbers are reproducible from make bench.
Storage backends
The default store.Store is store.NewMemory() — an in-memory flat cosine scan: exact, dependency-free, and fine into the ~10k-entry range.
For larger in-process indexes, store.NewHNSW() is a pure-Go (still zero-dependency) approximate-nearest-neighbour graph. On 20k×128-d vectors it is ~17× faster per lookup than the flat scan (≈0.3ms vs ≈5ms) at recall@10 ≈ 0.98:
c, _ := semcache.New(embedder, semcache.WithStore(store.NewHNSW()))
For a shared cache across processes or persistence, an optional RediSearch backend lives in a separate module so the root stays zero-dependency:
go get github.com/shironeko2707/semcache/store/redis
rs, _ := redisstore.New(ctx, redisstore.Config{Addr: "localhost:6379", Dim: 256})
c, _ := semcache.New(embedder, semcache.WithStore(rs))
It uses a RediSearch cosine index for KNN and Redis key TTLs for server-side expiry. See store/redis/. Any backend implementing store.Store works.
License
Apache-2.0. See LICENSE. Authored independently on personal time and hardware; see NOTICE.
Documentation
¶
Overview ¶
Package semcache is a correctness-aware semantic cache for LLM responses, built for regulated workloads where a wrong cache hit is a defect, not just a missed optimization. It pairs the usual embedding-similarity lookup with a similarity floor, a pluggable second-stage verifier, PII-aware keying, deterministic staleness via knowledge epochs, and a negative cache for near-misses. The default path is zero-dependency and in-memory.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNoEmbedder = errors.New("semcache: an Embedder is required")
ErrNoEmbedder is returned by New when no Embedder is supplied.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache interface {
Lookup(ctx context.Context, q Query) (Entry, bool, error)
Store(ctx context.Context, q Query, resp Entry) error
Stats() Stats
}
Cache is the public contract. Lookup returns a verified, non-stale, above-floor match or found=false (the caller should then call the model and Store the result).
type Embedder ¶
Embedder turns canonical query text into a vector. It is intentionally minimal and bring-your-own: wrap any provider (local model, hosted embedding API, ONNX runtime) behind this one method. semcache never locks you to a vendor and never calls a provider implicitly.
type EmbedderFunc ¶
EmbedderFunc adapts a plain function to the Embedder interface.
type FloorVerifier ¶
type FloorVerifier struct{}
FloorVerifier trusts the similarity floor alone and always accepts. Use only when false hits are cheap; not recommended for regulated workloads.
type HashEmbedder ¶
type HashEmbedder struct {
// contains filtered or unexported fields
}
HashEmbedder is a deterministic, dependency-free feature-hashing embedder. Each token is hashed to a dimension and a sign, accumulated, then the vector is L2-normalized. Lexically similar text yields similar vectors, which makes it ideal for tests, benchmarks, and offline/air-gapped development without a model. It is NOT a semantic model — swap in a real Embedder for production semantics. Provided here so the default path stays zero-dependency.
func NewHashEmbedder ¶
func NewHashEmbedder(dims int) *HashEmbedder
NewHashEmbedder returns a HashEmbedder producing dims-dimensional vectors. dims <= 0 falls back to 256.
type LexicalVerifier ¶
type LexicalVerifier struct {
MinOverlap float64 // Jaccard threshold in [0,1]; default 0.6 if zero
}
LexicalVerifier accepts only if the token Jaccard overlap between the query and the candidate's stored text meets MinOverlap. This cheaply rejects the classic semantic-cache failure mode where two queries are vector-close but differ in a decisive token (a negation, a different entity, a different amount) — exactly the case that hurts in banking.
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics holds lock-free counters for cache behaviour. The headline figure for the correctness thesis is FalseHitsAvoided: candidates that cleared the similarity floor but were rejected by verification — wrong answers the cache refused to serve. A generic cache never measures this.
type Option ¶
type Option func(*SemanticCache)
Option configures a SemanticCache.
func WithPolicy ¶
WithPolicy sets the correctness policy. Defaults to DefaultPolicy.
func WithRedactor ¶
WithRedactor sets the PII redactor. Defaults to NewRedactor.
func WithVerifier ¶
WithVerifier sets the second-stage verifier. Defaults to a LexicalVerifier.
type Policy ¶
type Policy struct {
// SimilarityFloor is the minimum cosine similarity for a candidate to even
// be considered. Necessary but not sufficient — verification still runs.
SimilarityFloor float32
// TTL bounds how long an entry may be served. Zero means no expiry.
TTL time.Duration
// EnforceEpoch requires the query's knowledge epoch to equal the entry's.
// A mismatch is always a miss, regardless of similarity — this is how
// answers that depend on changing facts expire deterministically.
EnforceEpoch bool
// EnforceMeta requires every key/value in the query's Meta to be present
// and equal in the candidate's Meta (tenant/filter isolation).
EnforceMeta bool
// NegativeCache records verified near-misses as negative anchors so a
// semantically-close-but-different query is not re-evaluated into a wrong
// hit and is short-circuited to a miss.
NegativeCache bool
// Candidates is how many nearest neighbours to verify before giving up.
Candidates int
}
Policy captures the correctness knobs that distinguish a regulated-workload cache from a generic one. Defaults are deliberately conservative: a high similarity floor and epoch/meta gating on, so the cache errs toward a miss (call the model) rather than risk a wrong hit.
func DefaultPolicy ¶
func DefaultPolicy() Policy
DefaultPolicy returns the recommended starting policy: a 0.92 floor, 24h TTL, epoch and meta gating enabled, negative cache on, top-5 candidate verification.
type Query ¶
type Query struct {
Text string // raw input; redacted internally before keying
Namespace string // tenant/collection isolation
Epoch string // knowledge epoch; mismatch => miss when enforced
Meta map[string]string // filters that must match for a hit when enforced
}
Query is a lookup or store request. Text is the raw user text; it is redacted and normalized internally before it is ever embedded, keyed, or stored.
type Redactor ¶
type Redactor struct {
// contains filtered or unexported fields
}
Redactor strips PII from query text before it is embedded, keyed, or stored, so secrets never enter the index. Patterns target synthetic Vietnamese PII shapes (CCCD/CMND, mobile numbers, bank cards/accounts, emails). Redaction runs before normalization. Order matters: more specific patterns first.
func NewRedactor ¶
func NewRedactor() *Redactor
NewRedactor returns a Redactor with the default Vietnamese-shaped rules.
func (*Redactor) AddRule ¶
AddRule appends a custom redaction rule. Custom rules run after the defaults.
func (*Redactor) Canonicalize ¶
Canonicalize redacts PII then normalizes the text (lowercase, collapse whitespace, trim). This canonical form is what gets embedded, keyed, and matched — never the raw input.
type SemanticCache ¶
type SemanticCache struct {
// contains filtered or unexported fields
}
SemanticCache is the default Cache implementation.
func New ¶
func New(embed Embedder, opts ...Option) (*SemanticCache, error)
New builds a SemanticCache. An Embedder is required; everything else has a sensible default (in-memory store, default policy, lexical verifier, default redactor). The metrics-wired eviction hook is attached only to the default store; a caller-supplied store should wire WithEvictHook itself if it wants eviction counted.
func (*SemanticCache) Lookup ¶
Lookup returns a cached response when a candidate clears the similarity floor, passes epoch/meta gating, is not a negative anchor, and passes verification. Otherwise found is false.
func (*SemanticCache) Stats ¶
func (c *SemanticCache) Stats() Stats
Stats returns a snapshot of the cache counters and current store size.
type Stats ¶
type Stats struct {
Hits uint64
Misses uint64
FalseHitsAvoided uint64
Stores uint64
Evictions uint64
Size int
}
Stats is an immutable snapshot of the counters plus the current store size.
func (Stats) FalseHitRate ¶
FalseHitRate estimates rejected-over-served-plus-rejected among floor-clearing candidates: FalseHitsAvoided / (Hits + FalseHitsAvoided). It approximates how often a floor-only cache would have served a wrong answer. Zero when no candidate ever cleared the floor.
type Verifier ¶
type Verifier interface {
// Verify reports whether candidate may be served for the given canonical
// query text. qText is already redacted+normalized.
Verify(ctx context.Context, qText string, candidate store.Record) (bool, error)
}
Verifier is the second stage of a hit decision. A high cosine score clears the similarity floor but is necessary, not sufficient: the Verifier decides whether the candidate is genuinely the same question. This is the core of the correctness-aware thesis — it is where false hits are caught before serving.
type VerifierFunc ¶
VerifierFunc adapts a function to the Verifier interface. Use this to plug in a cheap LLM-judge or any custom check. Kept off by default.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bench generates a synthetic, repetitive workload for measuring the semantic cache's hit rate and — crucially — its false-hit rate.
|
Package bench generates a synthetic, repetitive workload for measuring the semantic cache's hit rate and — crucially — its false-hit rate. |
|
Command gateway_middleware shows how an LLM gateway would put semcache in front of a model call — as a thin wrapper, with no import cycle and no provider lock-in.
|
Command gateway_middleware shows how an LLM gateway would put semcache in front of a model call — as a thin wrapper, with no import cycle and no provider lock-in. |
|
Package store defines the pluggable backend interface for semcache and ships a zero-dependency in-memory implementation.
|
Package store defines the pluggable backend interface for semcache and ships a zero-dependency in-memory implementation. |
|
redis
module
|