db

Documentation

Overview

Package db manages the local turso database that backs deadzone's vector-based semantic search. Documents are stored as TEXT alongside an F32_BLOB embedding column whose width is set at Open time from the embedder's reported Dim, and queries are ranked by vector_distance_cos against the query's embedding.

The package is intentionally embedder-agnostic: it does not import internal/embed, and the embedder's identity travels through the Meta struct supplied by the caller. The meta table records this identity in the database itself so a binary opening an existing file can detect (and refuse) a mismatch with the embedder it was asked to use.

Index

• Constants
• Variables
• func Bootstrap(ctx context.Context, appVersion string) (string, bool, error)
• func BootstrapWithOptions(ctx context.Context, opts BootstrapOptions) (string, bool, error)
• func DefaultCacheDir() string
• func EnableMultiProcessReaders()
• func Insert(db *DB, doc Doc, embedding []float32) error
• func UpdateLibCount(d *DB, libID, version string, count int) error
• func UpsertLibIfNew(d *DB, libID, version, description string, e LibEmbedder) error
• type ArtifactMeta
  • func ReadArtifactMeta(path string) (ArtifactMeta, error)
• type BootstrapOptions
• type ConsolidateResult
  • func Consolidate(main *DB, artifactsDir string) (ConsolidateResult, error)
• type DB
  • func Open(path string, meta Meta) (*DB, error)
  • func OpenArtifact(path string, meta Meta, libID, version string) (*DB, error)
  • func OpenReader(path string, meta Meta) (*DB, error)
• type Doc
  • func SearchByEmbedding(db *DB, queryVec []float32, libID, version string, k int) ([]Doc, error)
• type LibEmbedder
• type LibInfo
  • func SearchLibsByEmbedding(d *DB, queryVec []float32, limit int) ([]LibInfo, error)
  • func TopLibsByDocCount(d *DB, limit int) ([]LibInfo, error)
• type Meta

Constants

const (
	EnvCacheDir = "DEADZONE_DB_CACHE"
	EnvOffline  = "DEADZONE_DB_OFFLINE"
	// EnvAutoUpdate gates the boot-time freshness probe added in #197.
	// Default (unset, or any non-"0"/"false" value) is enabled.
	// Setting "0" / "false" opts the implicit server-boot path out;
	// `fetch-db` still probes regardless because the env var only
	// governs the implicit path — explicit `fetch-db` is by definition
	// a user-driven refresh.
	EnvAutoUpdate = "DEADZONE_DB_AUTOUPDATE"

	BootstrapDefaultRepo = "laradji/deadzone"

	// DevVersion is the sentinel AppVersion for local / unreleased
	// builds. Bootstrap falls back to /releases/latest on it.
	DevVersion = "dev"
)

Cache / asset constants. The asset names match what `deadzone dbrelease` uploads; changing either side requires changing both.

const CurrentSchemaVersion = 5

CurrentSchemaVersion is the on-disk schema version written by this build. Bump whenever the table layout changes in a non-backwards- compatible way OR when the embedding semantics change (so old DB vectors and new query vectors live in different spaces — even if the table layout is identical, the rows are stale). Stored in the meta table at first Open and cross-checked on every subsequent open against this constant; a mismatch surfaces as ErrSchemaMismatch.

Bump history:

• 1: initial layout
• 2: pluggable embedder metadata (#84)
• 3: libs catalog table (#55)
• 4: first-class multi-version (#113)
• 5: lib embedding mixes optional description with lib_id (#191)

const EnvDisableFileLock = "LIMBO_DISABLE_FILE_LOCK"

EnvDisableFileLock is the tursogo (Limbo) escape hatch that makes every IO backend skip the OS-level fcntl lock at file-open time. Set to any non-empty value before the first OpenReader call. Defined in tursogo's core/io/common.rs as ENV_DISABLE_FILE_LOCK.

Variables

var ErrArtifactLibIDMismatch = errors.New("artifact lib_id mismatch")

ErrArtifactLibIDMismatch is returned by OpenArtifact when both the stored and the requested lib_id are non-empty but disagree. Catches the failure mode where an artifact gets renamed on disk so its filename and recorded lib_id no longer match, which would otherwise silently merge rows under the wrong key.

var ErrArtifactLibIDMissing = errors.New("artifact has no lib_id in meta")

ErrArtifactLibIDMissing is returned by OpenArtifact when the caller passes libID == "" (i.e. asks the artifact to identify itself) but the on-disk meta table has no lib_id key. Wrap with errors.Is to detect — the consolidate path treats it as a structural problem with the artifact file itself, not a transient I/O error.

var ErrEmbedderMismatch = errors.New("embedder metadata mismatch")

ErrEmbedderMismatch is returned by Open when the meta stored in an existing database disagrees with the Meta the caller passed in. Callers should treat this as fatal: there is no safe way to mix vectors produced by different embedders in the same docs table. Wrap with errors.Is to detect.

var ErrNoReleaseForVersion = errors.New("no deadzone.db published for this binary version")

ErrNoReleaseForVersion is returned when the requested AppVersion has no corresponding release on GitHub. Callers (main, tests) check via errors.Is so the user-facing error message can be precise. Wraps nothing — it IS the error.

var ErrReaderBusy = errors.New("another process is using this database file")

ErrReaderBusy is returned by OpenReader when another process already holds the OS-level file lock on the database file. Surfaces the otherwise-cryptic tursogo "Locking error: Failed locking file" with a typed sentinel so callers (e.g. cmd/deadzone/server.go) can print a human-readable message naming the offending file. See docs/research/multi-process-lock.md for why this is needed and what it costs to detect. Wrap with errors.Is.

var ErrReaderNotInitialized = errors.New("database was never initialized by a mutator")

ErrReaderNotInitialized is returned by OpenReader when the file exists but has no meta rows — i.e. a mutator has never opened it to lay down the schema. Readers must not bootstrap DBs themselves (that is exclusively the mutator path's job), so the right fix is to run a mutator command (consolidate / dbrelease / scrape) first. Wrap with errors.Is to detect.

var ErrSchemaMismatch = errors.New("database schema version mismatch")

ErrSchemaMismatch is returned by Open when the database's stored schema version does not match CurrentSchemaVersion. Callers should treat this as fatal: the current code cannot read a database produced by a build with a different table layout. Use a fresh database file (drop and re-scrape) until an in-place migration is implemented. Wrap with errors.Is to detect.

Functions

func Bootstrap

func Bootstrap(ctx context.Context, appVersion string) (string, bool, error)

Bootstrap ensures a deadzone.db is present at the default cache location, pinned to appVersion, and returns its path. See the file banner for the full contract.

Returns (path, upgraded, error) where upgraded is true when this call just replaced an existing cache file (version bump or -force). First-fetch returns upgraded=false (no previous file existed).

func BootstrapWithOptions

func BootstrapWithOptions(ctx context.Context, opts BootstrapOptions) (string, bool, error)

BootstrapWithOptions is the explicit-options form of Bootstrap. fetch-db uses it for -force and -repo; tests use it to inject a CacheDir without touching env vars.

func DefaultCacheDir

func DefaultCacheDir() string

DefaultCacheDir resolves the cache root used by Bootstrap when the caller passes an empty CacheDir / DEADZONE_DB_CACHE is unset.

Resolution order:

1. $DEADZONE_DB_CACHE if set.
2. Platform data dir (per-platform, see below).
3. ./.deadzone-cache/db as a last-resort fallback so Bootstrap can still proceed when the home dir lookup fails.

Per-platform data dirs:

• macOS: ~/Library/Application Support/deadzone
• Linux: $XDG_DATA_HOME/deadzone (falls back to ~/.local/share/deadzone)
• Windows: %LOCALAPPDATA%\deadzone (falls back to ~/AppData/Local/deadzone)

We don't reuse os.UserCacheDir because the issue intentionally sites the DB under the persistent data dir, not the volatile cache dir. The DB is the user's installation, not a regenerable cache.

func EnableMultiProcessReaders

func EnableMultiProcessReaders()

EnableMultiProcessReaders opts the current process into the lock bypass so multiple readers can coexist on the same database file. Safe to call repeatedly. Intended for callers (e.g. deadzone server) that only ever use OpenReader: the bypass weakens cross-process protection against concurrent writers, so processes that take the mutator path (Open, OpenArtifact) must NOT call this.

Why an env var rather than an open flag: tursogo v0.5.3 does not expose OpenFlags::ReadOnly through its DSN or config struct, so the env-var escape hatch is the only Go-reachable lever. See docs/research/multi-process-lock.md for the full audit and the migration plan once tursogo ships native multi-process support.

func Insert

func Insert(db *DB, doc Doc, embedding []float32) error

Insert stores a Doc along with its precomputed embedding. The embedding must have exactly db.Meta.EmbeddingDim components — the dimension travels with the *DB rather than being a package-level constant so a single binary can support multiple embedder kinds.

func UpdateLibCount

func UpdateLibCount(d *DB, libID, version string, count int) error

UpdateLibCount sets the doc_count for an existing libs row keyed on (libID, version). The scraper calls this once per lib at the end of a run with the actual number of docs that were inserted, so search_libraries can surface "how well-indexed is this lib" without recounting on every query. Updating a row that does not exist is silently a no-op (zero rows affected) — the scraper is responsible for calling UpsertLibIfNew first.

func UpsertLibIfNew

func UpsertLibIfNew(d *DB, libID, version, description string, e LibEmbedder) error

UpsertLibIfNew inserts a row into the libs table for (libID, version) iff one doesn't already exist. The embedding text is built by libEmbedText: the normalized lib_id ("/hashicorp/terraform-provider-aws" → "hashicorp terraform provider aws") concatenated with the upstream- authored description (when non-empty) so the encoder sees both the canonical name tokens and the intent prose. Empty description falls back to lib_id alone — the legacy embedding path, byte-for-byte the same vector pre-#191.

Version is still NOT mixed into the embed text: the version identifier is a user-facing label with no semantic value to a free-text library-name query. Description, on the other hand, CAN differ per version when a per-version override is set (terraform 1.13 vs 1.14 with a major API rewrite), in which case two versions of the same lib_id WILL produce distinct embeddings. When descriptions match (or both are empty) the legacy "all versions of a lib rank identically" property is preserved for the common case.

Re-running this function for an existing (lib_id, version) pair is a fast no-op that does NOT call EmbedDocument — the issue's "at most one Embed call per (lib, version) per database" guarantee is enforced here and verified by tests that count the call against a wrapping LibEmbedder.

Version is the canonical empty string for single-version libs; the primary key is on (lib_id, version) so the same base lib with two versions cleanly produces two rows.

Types

type ArtifactMeta

type ArtifactMeta struct {
	LibID        string
	EmbedderKind string
	ModelVersion string
}

ArtifactMeta is the identity payload of a per-lib artifact: which library it carries and which embedder produced its vectors. The per-pack manifest committed to git records exactly these three fields alongside the sha256, so a future cmd/packs upload can recreate the manifest entry from a freshly-built artifact without having to spin up an embedder.

func ReadArtifactMeta

func ReadArtifactMeta(path string) (ArtifactMeta, error)

ReadArtifactMeta opens path with a bare sql.Open, reads the three identity keys from the meta table, and closes. Unlike OpenArtifact it does NOT validate the embedder meta against a Meta passed by the caller and does NOT cross-check the schema version, so callers can inspect an artifact without instantiating an embedder (and paying the ~90MB MiniLM model download on first run).

The trade-off: ReadArtifactMeta only tells you what the artifact claims to be. The integrity contract that those vectors actually match the embedder is enforced exactly once, by db.Open / OpenArtifact, at the moment a real consumer (consolidate, server) loads the file. cmd/packs is intentionally upstream of that check — it just shovels bytes around.

Returns ErrArtifactLibIDMissing when the file exists but has no lib_id row, mirroring OpenArtifact's behaviour for the same case so callers using errors.Is don't have to learn a second sentinel.

type BootstrapOptions

type BootstrapOptions struct {
	// CacheDir overrides DEADZONE_DB_CACHE / the platform default.
	CacheDir string
	// Repo overrides BootstrapDefaultRepo (owner/name).
	Repo string
	// AppVersion is the binary's own version string (e.g. "v0.1.0"
	// from the justfile's -ldflags). Required in practice; an empty
	// string is treated as DevVersion and triggers the /latest
	// fallback.
	AppVersion string
	// Force re-fetches even when the cached tag matches AppVersion.
	// fetch-db --force uses this; the server path leaves it false so
	// a no-op startup stays zero-network.
	Force bool
	// SkipAutoUpdateProbe disables the boot-time freshness probe added
	// in #197 even when the cache is hot and online. The server path
	// sets this from DEADZONE_DB_AUTOUPDATE=0; fetch-db NEVER sets it
	// (the probe is precisely what `fetch-db` is for, no-flag).
	SkipAutoUpdateProbe bool
	// Caller is a free-form label that flows into the structured logs
	// the auto-update probe emits (`db.update_*` events). The two
	// production callers are "server" and "fetch-db"; tests leave it
	// empty.
	Caller string
}

BootstrapOptions lets callers (runServer, runFetchDB, tests) override what Bootstrap pulls from env / defaults.

type ConsolidateResult

type ConsolidateResult struct {
	// Artifacts is the number of artifact files merged into main.
	Artifacts int
	// DocsMerged is the total number of docs rows inserted into main
	// across every merged artifact (i.e. the new row count for the
	// libs that participated; rows displaced by the per-lib DELETE
	// are not counted as "merged").
	DocsMerged int
	// LibsMerged is the number of libs rows inserted into main. Each
	// artifact contributes either 0 (degenerate empty artifact) or 1.
	LibsMerged int
}

ConsolidateResult is the per-run summary returned by Consolidate. The counts are best-effort accounting for the operator log line; they have no bearing on the on-disk state, which is governed entirely by the transaction's commit/rollback.

func Consolidate

func Consolidate(main *DB, artifactsDir string) (ConsolidateResult, error)

Consolidate merges every per-lib artifact in artifactsDir into main. Each lib lives in its own subdirectory (artifacts/<slug>/artifact.db + state.yaml, see #101), so consolidation globs the nested shape rather than the old flat *.db layout. Artifacts are keyed on (lib_id, version) after #113 — two versions of the same base lib live in two sibling folders and merge as two independent (lib_id, version) slots without clobbering each other. Each artifact replaces (not appends to) the rows in main for its (lib_id, version) slot, in both the docs and libs tables, atomically.

The operation runs in two passes:

1. Validation. Every artifact is opened with main's Meta so any embedder mismatch (ErrEmbedderMismatch), schema mismatch (ErrSchemaMismatch), or missing artifact lib_id (ErrArtifactLibIDMissing) surfaces *before* any write touches main. Failures here leave main byte-identical.

2. Merge. A single transaction is begun on main; for each validated artifact, the existing rows for its lib_id are deleted from docs and libs, and the artifact's rows are streamed in via the transaction. The transaction commits at the end of the loop. If any step in pass 2 fails, the transaction rolls back and main is restored to its pre-call state.

Reading and writing happen on different *sql.DB connection pools (one per artifact, one for main), so the merge does not contend with the per-pool MaxOpenConns=1 cap that tursogo serializes on.

Artifact files are processed in lexicographic order so that the merged-on-disk doc IDs are stable across runs — useful for debugging diff'ing the same corpus on two machines.

type DB

type DB struct {
	*sql.DB
	Meta            Meta
	ArtifactLibID   string
	ArtifactVersion string
}

DB wraps *sql.DB with the Meta the database was opened with so that Insert and SearchByEmbedding can validate vector lengths without re-reading the meta table on every call. *sql.DB is embedded so callers can still use QueryRow, Exec, Close, etc. directly on a *DB.

ArtifactLibID / ArtifactVersion are populated only when the database was opened via OpenArtifact. Together they form the canonical (lib_id, version) slot this artifact carries (read from the meta table at open time). ArtifactVersion is "" for single-version libs, matching the on-wire canonical form. The main consolidated database always leaves both empty — the libs table is the source of truth there.

func Open

func Open(path string, meta Meta) (*DB, error)

Open opens (or creates) a local turso database at path and ensures the schema is in place. The meta argument describes the embedder the caller intends to use:

• On a fresh database, the meta is persisted and the docs table is created with an F32_BLOB column whose width matches meta.EmbeddingDim.
• On an existing database, the stored meta must equal the supplied meta — otherwise ErrEmbedderMismatch is returned, wrapped with the conflicting values so the user knows what to do (typically: rebuild the database with a fresh file).

tursogo's DSN is a bare path — the "file:" prefix used by libSQL is NOT stripped and would create a file literally named "file:<path>".

func OpenArtifact

func OpenArtifact(path string, meta Meta, libID, version string) (*DB, error)

OpenArtifact opens (or creates) a per-(lib_id, version) artifact database. An artifact carries a single (lib_id, version) pair recorded in its meta table; the recorded values are the source of truth for which library slot the artifact's docs and libs rows belong to.

(libID, version) semantics:

• libID != "" — the caller knows which (lib_id, version) slot this artifact represents (e.g. the scraper). On a fresh file the pair is written. On an existing file both values must match the stored ones, otherwise ErrArtifactLibIDMismatch is returned. Version may be "" for single-version libs — that is the canonical form.

• libID == "" — the caller is reading an existing artifact and wants to discover its (lib_id, version) (e.g. consolidate). Version must also be "" in this mode; it is populated from the stored meta. The file must already exist; if it doesn't, an os.ErrNotExist-wrapped error is returned without creating a stub. If the file exists but has no lib_id stored, ErrArtifactLibIDMissing is returned.

Embedder meta and schema version validation are inherited from Open — an artifact built with a different embedder than the caller's surfaces as ErrEmbedderMismatch, exactly like the main DB.

func OpenReader

func OpenReader(path string, meta Meta) (*DB, error)

OpenReader opens an existing deadzone database in read-only mode. It is the entry point used by `deadzone server` and any other caller that issues only SELECTs. Unlike Open it does NOT run any DDL and does NOT accept writes: a PRAGMA query_only = 1 is set on the connection immediately after open so any subsequent INSERT / UPDATE / DELETE / CREATE TABLE returns a SQLite "attempt to write a readonly database" error.

Motivation: Open's unconditional CREATE TABLE IF NOT EXISTS meta takes a write-intent lock on every boot, which serializes concurrent MCP server processes against the same deadzone.db file. OpenReader skips all schema DDL so N reader processes can coexist without SQLITE_BUSY, while preserving the usual schema + embedder meta cross-check semantics.

Contract vs Open:

• Same validateMeta / schema_version / embedder meta checks. ErrSchemaMismatch and ErrEmbedderMismatch surface exactly as they do from Open, so callers can errors.Is them identically.
• The file MUST exist. If it does not, os.ErrNotExist is returned wrapped with the path — readers never fabricate empty stubs.
• The file MUST have been initialized by a mutator: if the meta table is absent or empty, ErrReaderNotInitialized is returned.
• Writes fail fast via query_only and (as a second line of defence) the caller-facing *DB has no helper that attempts mutations.

type Doc

type Doc struct {
	LibID   string
	Version string
	Title   string
	Content string
}

Doc represents a documentation snippet stored in the docs table.

Version is the canonical form for single-version libs: empty string, not NULL. Multi-version libs pass the version tag as recorded in the registry (e.g. "v1.14"). LibID always carries the base lib_id — the "/base/version" concat that earlier builds emitted is gone.

func SearchByEmbedding

func SearchByEmbedding(db *DB, queryVec []float32, libID, version string, k int) ([]Doc, error)

SearchByEmbedding returns the top-k Docs ranked by cosine distance to queryVec (lower = more similar). k defaults to 10 if <= 0. The query vector must have db.Meta.EmbeddingDim components.

The (libID, version) filter is three-valued:

• libID == "" — no filter (version is ignored).
• libID != "" and version == "" — match every indexed version of the lib (WHERE lib_id = ?). The single-version case is the same query since those rows carry version = "".
• libID != "" and version != "" — pin to a specific version (WHERE lib_id = ? AND version = ?).

Passing version without libID is a usage error at the MCP tool layer; this function does not enforce it so unit tests stay small.

type LibEmbedder

type LibEmbedder interface {
	EmbedDocument(text string) ([]float32, error)
}

LibEmbedder is the minimal subset of an embedder that UpsertLibIfNew needs. Defining a local interface keeps the db package free of any import on internal/embed (matching the package-level rule the docs table already follows) while still letting tests pass a counting wrapper that asserts the embed call is actually skipped on the idempotent re-upsert path.

EmbedDocument (not EmbedQuery) is the right method here: a lib_id row is indexed content, retrieved against user queries via SearchLibsByEmbedding. Retrieval-trained models use different prefixes for the two sides and mixing them up silently degrades match quality.

type LibInfo

type LibInfo struct {
	LibID    string
	Version  string
	DocCount int
	Distance float32
}

LibInfo is one row of the libs table as returned by SearchLibsByEmbedding and TopLibsByDocCount. Distance is the raw cosine distance from the query vector (lower is better, 0 is identical, 2 is maximally far); callers convert it to a 1 - dist match score before serializing to the wire so the LLM sees a monotonically-good number. The doc-count path returns Distance = 0 (no query was issued), and the search path fills it in from vector_distance_cos.

func SearchLibsByEmbedding

func SearchLibsByEmbedding(d *DB, queryVec []float32, limit int) ([]LibInfo, error)

SearchLibsByEmbedding returns the top-`limit` libs ranked by cosine distance to queryVec, breaking ties by doc_count desc so a well-indexed lib outranks a barely-touched one when both score equally on semantic match. The query vector must have db.Meta.EmbeddingDim components — the same constraint as SearchByEmbedding, for the same reason (cross-embedder vectors are nonsense).

func TopLibsByDocCount

func TopLibsByDocCount(d *DB, limit int) ([]LibInfo, error)

TopLibsByDocCount returns the top-`limit` libs ranked by doc_count descending. This is the cheap "no query" path that powers the empty- name branch of search_libraries: an LLM exploring an unfamiliar corpus gets a useful "what's even in here" answer without paying for an embedder call. Distance is left at 0 (the row's match_score in the wire format ends up at 1.0).

type Meta

type Meta struct {
	EmbedderKind string
	EmbeddingDim int
	ModelVersion string
}

Meta describes the embedder a database was created with. It is written to the meta table the first time a fresh DB is opened and cross-checked on every subsequent open.

Equality is by value: every field must match exactly for a reopen to be accepted. Bumping ModelVersion in the embedder is therefore the standard way to invalidate previously-indexed databases.