semantic

Documentation

Overview

Package semantic is the optional vector channel: a float32-BLOB VectorStore living in the SAME cache db as the keyword index (under its own schema gate), brute-force cosine KNN in plain Go, and reciprocal-rank fusion.

It is NEVER in the keyword _rebuild() drop list and never bumps the keyword SchemaVersion — a keyword reindex can't nuke vectors and vice versa. Vectors are keyed by (session_id, content_hash) so msg-id churn on reindex is harmless.

Vectors are packed as little-endian float32 BLOBs. The chunk_vec SQL itself lives in the store package (D4) — semantic keeps hashing, embedding, cosine KNN, and RRF fusion.

Index

• Constants
• func AcquireTopupToken(dbp string, now time.Time) bool
• func Fuse(con *sql.DB, kwRows []retrieve.Anchor, qvec []float64, knnK int, ...) []retrieve.Anchor
• func IsNoVector() bool
• func MaybeVectorTopup(dbp string)
• func SetNoVector(v bool)
• func SetSpawnVectorTopup(fn func(string))
• func TryAcquireTopupLock(dbp string) (release func(), ok bool)
• func VecIndex(ctx context.Context, con *sql.DB, embedder embed.Embedder, maxNew int) (added int, err error)
• func VectorTopupLogPath() string
• type CoverageStats
  • func MeasureCoverage(con *sql.DB, projects ...string) (CoverageStats, error)
• type VecHit
  • func VecKNN(con *sql.DB, qvec []float64, k int, includeSubagents bool) []VecHit

Constants

const DefaultTopupMaxNew = 50

DefaultTopupMaxNew is the maximum number of new messages embedded during one background top-up pass. Bounding per-invoke work prevents an un-embedded or newly imported corpus from stampeding the embedder backend or locking the db.

const MinChars = 12

MinChars skips trivial messages (greetings, acks).

const RRFConstant = 60

RRFConstant is the reciprocal-rank-fusion k.

Functions

func AcquireTopupToken

func AcquireTopupToken(dbp string, now time.Time) bool

AcquireTopupToken reports whether a background vector top-up may spawn now for dbp, and atomically claims the slot when it may. Mirrors archive.AcquireAutosyncToken.

func Fuse

func Fuse(con *sql.DB, kwRows []retrieve.Anchor, qvec []float64, knnK int, includeSubagents bool) []retrieve.Anchor

Fuse merges keyword anchors + vector KNN via RRF(k=RRFConstant) by message id, returning a merged anchor list ordered by fused score (each row's Fused set). Keyword-only rows keep their fields; vector-only rows are synthesized (Cov=0).

func IsNoVector

func IsNoVector() bool

IsNoVector reports whether --no-vector is active.

func MaybeVectorTopup

func MaybeVectorTopup(dbp string)

MaybeVectorTopup quietly keeps a scope's vectors current after an ordinary indexing pass. Detached self-invocation so search latency is unaffected. Gates: 1. IsNoVector() returns true -> return (respect --no-vector). 2. adapters.GetEmbedder() returns nil -> return (unconfigured, zero spawns). 3. AcquireTopupToken(dbp, now) -> rate-limit spawns per store.

func SetNoVector

func SetNoVector(v bool)

SetNoVector records whether --no-vector is active for the current process/invocation.

func SetSpawnVectorTopup

func SetSpawnVectorTopup(fn func(string))

SetSpawnVectorTopup sets the spawner hook (wired by package cli to spawn a detached child).

func TryAcquireTopupLock

func TryAcquireTopupLock(dbp string) (release func(), ok bool)

TryAcquireTopupLock takes the machine-wide single-writer lock for vector top-up on dbp WITHOUT waiting. ok=false if another top-up is already running on dbp or on any lock error. Uses flock.New, mirroring archive's tryAcquireSyncLock.

func VecIndex

func VecIndex(ctx context.Context, con *sql.DB, embedder embed.Embedder, maxNew int) (added int, err error)

VecIndex embeds any message (tool-stripped prose ≥ MinChars) not yet vectored, prunes stale vectors, and refreshes churned msg_ids. Resumable; returns the count added. `maxNew` of 0 = no cap. ctx bounds the embedding pass. It is load-bearing rather than decorative: the batch path fans out to several goroutines each blocked on a remote HTTP call, and without a cancellation signal a Ctrl-C leaves them running to completion against the endpoint while the caller has already gone. Every vector committed before cancellation stays committed — the pass is resumable, so a cancelled run costs only the batch in flight.

func VectorTopupLogPath

func VectorTopupLogPath() string

VectorTopupLogPath is <state-dir>/vector-topup.log — where the detached vector top-up child's output lands.

Types

type CoverageStats

type CoverageStats struct {
	Candidates int
	Vectored   int
	Missing    int
}

CoverageStats reports candidate embedding coverage across a store.

func MeasureCoverage

func MeasureCoverage(con *sql.DB, projects ...string) (CoverageStats, error)

MeasureCoverage scans live candidate messages (tool-stripped prose >= MinChars) and compares them against stored vectors in chunk_vec. If projects is provided, only messages belonging to those projects are considered.

type VecHit

type VecHit struct {
	ID            int
	SessionID     string
	ISO           string
	Parent        string
	OnlyCopySince float64 // sessions.only_copy_since (0 when NULL); carried so a vector-only hit on an only-copy session keeps the flag
	Dist          float64 // cosine similarity (higher = nearer)
}

VecHit is one vector-KNN anchor: id, session_id, iso, parent, dist.

func VecKNN

func VecKNN(con *sql.DB, qvec []float64, k int, includeSubagents bool) []VecHit

VecKNN returns up to k vector-anchor VecHits nearest to qvec, existence-checked against the live messages table.