Documentation
¶
Overview ¶
Package embed produces fixed-dimension embedding vectors from text.
This package defines the Embedder interface used by both the indexer (cmd/scraper) and the query path (cmd/server). The current and only implementation is Hugot, a feature extraction pipeline running on hugot's ORT (onnxruntime) backend (see hugot.go).
The interface keeps room for future embedders behind the same factory: adding a second model family means adding a Kind constant and a New case, not changing every call site.
Index ¶
Constants ¶
const DefaultHugotModel = "nomic-ai/nomic-embed-text-v1.5"
DefaultHugotModel is nomic-embed-text-v1.5 — 768-dim, 8192-token context, Apache-2.0. Replaces the earlier all-MiniLM-L6-v2 checkpoint that panicked on inputs above 512 tokens and only produced 384-dim embeddings. Bumping this constant invalidates every existing database via db.Meta cross-check on next Open.
const KindHugot = "hugot"
KindHugot is the Kind() value reported by the Hugot embedder, and the only kind currently accepted by New.
const ModelRevision = "e9b6763023c676ca8431644204f50c2b100d9aab"
ModelRevision is the HuggingFace commit SHA the pinned files were fetched from. It is the model-side equivalent of internal/ort.Version — the operator-facing identifier of "which exact bytes are baked in this image." Captured from the HuggingFace API:
curl -fsSL https://huggingface.co/api/models/nomic-ai/nomic-embed-text-v1.5 | jq -r .sha
at the time the bake was authored (2026-05-05). HF commit SHAs are 40-hex-char immutable refs — point-in-time snapshots, not branches — so this revision will resolve to the same bytes forever as long as the model repository remains public.
Variables ¶
This section is empty.
Functions ¶
func DefaultCacheDir ¶
func DefaultCacheDir() string
DefaultCacheDir returns the cache directory used by NewHugot when the caller passes an empty cacheDir, and by embed.New for the same purpose.
Resolution order:
- $DEADZONE_HUGOT_CACHE if set (used by CI to pin the cache to a workspace-local path that persists across runs, and by users who want to share the model cache across processes or place it on a specific disk).
- os.UserCacheDir() + /deadzone/models — the platform default: - Linux: $XDG_CACHE_HOME/deadzone/models (or ~/.cache/deadzone/models) - macOS: ~/Library/Caches/deadzone/models - Windows: %LOCALAPPDATA%\deadzone\models
- ./.deadzone-cache/models as a last-resort fallback when UserCacheDir fails so the constructor can still proceed.
Exported (capitalized) so tests in this and other packages can call the same resolution logic instead of duplicating the env-var check.
func HugotSignature ¶ added in v0.6.0
func HugotSignature() string
HugotSignature returns a deterministic sha256-hex digest over every hugot.go constant that affects the vector space the Hugot embedder produces. Used as a CI cache-invalidation signal in scrape-pack.yml so that bumping any one of these constants forces a corpus rescrape in lockstep — independently of pure-refactor edits to hugot.go that don't change the vector space (which a `hashFiles(hugot.go)` cache key would have over-invalidated on).
The function is package-level (no Embedder instance) so CI can call it via `deadzone cache-signals` BEFORE the model download / ORT init steps. Loading the actual model just to compute its identity would defeat the cache layer's whole purpose.
Adding a new vector-space-affecting constant later (e.g. a pooling mode) means appending it to the parts slice here. The pinned snapshot in TestHugotSignature_Stable fails on any digest change, forcing the contributor to either confirm the cache invalidation is intended or to roll back the change.
func ModelDestDirname ¶ added in v0.7.0
func ModelDestDirname() string
ModelDestDirname is the basename of the directory hugot expects the model files to live in, computed from DefaultHugotModel via the same "/" → "_" rule hugot's downloader applies (hugot.go:127). Exposed so the OCI image build and the hidden hugot-meta subcommand share the same derivation rather than each reproducing the rule and drifting on a future model swap.
func Signature ¶ added in v0.6.0
Signature returns the deterministic vector-space signature for the given embedder kind WITHOUT instantiating it (no model load, no network). Mirrors New(kind) — same valid-kinds set, same error shape on unknown kinds. Used by `deadzone cache-signals` to feed the CI per-lib artifact cache key, so a constant edit in hugot.go (model swap, quantization variant change, prefix tweak) invalidates caches in lockstep without forcing a Hugot init at workflow time.
Types ¶
type Embedder ¶
type Embedder interface {
// EmbedQuery returns a vector of length Dim() for a retrieval query —
// text that will be compared against a corpus. Implementations must
// surface inference errors instead of returning a placeholder vector:
// a silently corrupted embedding pollutes the cosine index permanently
// and is impossible to detect post-hoc.
EmbedQuery(text string) ([]float32, error)
// EmbedDocument returns a vector of length Dim() for a corpus
// document — text that will live in the index (a scraped snippet, a
// lib_id row, …). Same error-propagation contract as EmbedQuery.
EmbedDocument(text string) ([]float32, error)
// Kind identifies the embedder family (e.g. "hugot").
// Used for meta consistency checks between scraper and server runs.
Kind() string
// Dim is the output vector dimension. Must be constant for the
// lifetime of the Embedder.
Dim() int
// ModelVersion identifies the specific model producing the
// embeddings (e.g. "nomic-ai/nomic-embed-text-v1.5"). Stored in the
// DB meta table and cross-checked at open time.
ModelVersion() string
// Close releases any resources held by the embedder (model session,
// tokenizer, etc.). Safe to call once at process shutdown via defer.
Close() error
}
Embedder turns text into a fixed-dimension embedding vector. Implementations must be deterministic: the same input must always produce the same output, so that indexed vectors and query vectors are comparable.
The interface is split into EmbedQuery and EmbedDocument because retrieval-trained models (nomic-embed-text, BGE, e5, …) require task-specific prefixes and degrade silently without them. The call site knows whether a given text is a query or a document; the embedder picks the right prefix accordingly. Implementations that do not need a prefix can forward both methods to a single internal path.
Kind, Dim, and ModelVersion are written into the database's meta table on first use and cross-checked on every subsequent open, so that a binary running with embedder X cannot accidentally read or write a database that was indexed with embedder Y.
func New ¶
New returns an Embedder for the given kind. Currently only KindHugot is supported; unknown kinds return an error so cmd/scraper and cmd/server can fail cleanly with a useful message.
The Hugot embedder uses DefaultHugotModel and the platform default cache directory. Callers needing a non-default model or cache location should call NewHugot directly.
type Hugot ¶
type Hugot struct {
// contains filtered or unexported fields
}
Hugot wraps a hugot Session + FeatureExtractionPipeline running on the ORT (onnxruntime) backend. One Hugot is meant to live for the lifetime of a process: NewHugot is expensive (downloads + loads the model + spins up the ORT environment) but each EmbedQuery / EmbedDocument call is cheap.
Concurrency: hugot's pipelines are not documented as goroutine-safe. internal/db serializes its single connection, and cmd/server handles one MCP request at a time, so a single shared *Hugot is fine for the current workload. If parallelism is added later, wrap the embed calls in a mutex or pool pipelines per worker.
func NewHugot ¶
NewHugot constructs a Hugot embedder backed by modelName, downloading the model into cacheDir if it isn't already present. Pass an empty modelName to use DefaultHugotModel.
First-run cost is dominated by the ONNX model download (~131 MB for the int8 nomic quantized variant) and the ORT session warm-up. Subsequent runs reuse the on-disk model.
The ORT shared library is resolved via internal/ort.Bootstrap: the pinned release is downloaded + SHA256-verified + cached on first use and re-used on every subsequent run. Set DEADZONE_ORT_LIB_PATH to bypass the download and point at a hand-positioned library (air-gapped installs, pinned mirrors). Building with `-tags ORT` and CGO_ENABLED=1 is required — without the tag, hugot.NewORTSession below compiles as a stub that returns a clear error.
func (*Hugot) Close ¶
Close releases the underlying hugot Session and every pipeline owned by it. Safe to call once at process shutdown via defer.
func (*Hugot) Dim ¶
Dim is the output vector dimension, discovered from the loaded model at construction time. 768 for the default nomic checkpoint.
func (*Hugot) EmbedDocument ¶
EmbedDocument prepends the "search_document: " prefix that nomic was trained with for indexed passages and returns the resulting unit-norm vector. Use this when embedding content that will live in the corpus (e.g. a scraped doc, or a lib_id written into the libs table).
func (*Hugot) EmbedQuery ¶
EmbedQuery prepends the "search_query: " prefix that nomic was trained with for retrieval queries and returns the resulting unit-norm vector. Use this for text that will be compared against a corpus (e.g. a user query handed to search_docs or search_libraries).
func (*Hugot) ModelVersion ¶
ModelVersion returns the Hugging Face model name. Used by db.Meta to invalidate databases when the user switches models — bumping the model triggers ErrEmbedderMismatch on next Open against an old database.
type PinnedModelFile ¶ added in v0.7.0
type PinnedModelFile struct {
URL string `json:"url"`
SHA256 string `json:"sha256"`
DestName string `json:"dest_name"`
}
PinnedModelFile is the public-facing version of pinnedModelFile, returned by PinnedModel. Field names are stable JSON contract surface — cmd/deadzone/hugot_meta.go marshals these directly and scrape-pack.yml's docker job parses the JSON with jq. A breaking change here breaks the OCI image build before it can fail loud at `docker buildx build` time.
func PinnedModel ¶ added in v0.7.0
func PinnedModel() (modelID, revision, destDirname string, files []PinnedModelFile)
PinnedModel returns the metadata an out-of-band stager needs to populate the hugot cache without invoking the embedder runtime:
- modelID: the HuggingFace model name (DefaultHugotModel).
- revision: the pinned commit SHA (ModelRevision).
- destDirname: the directory name hugot expects under the cache root (ModelDestDirname()).
- files: one entry per file with the absolute download URL, pinned SHA256, and target filename inside destDirname.
Mirrors the shape of internal/ort.PinnedRelease — same out-of-band stager use case, same "constants table is the contract" philosophy.
Source Files