embed

package
v0.0.0-...-628cd63 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 5, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package embed wires the engram embedder, sidecar format, and staleness detection for the embed-on-write semantic-search pipeline.

Index

Constants

View Source 
const (
	// AnsweredByBodyMarker prefixes the machine-written `Answered by: [[…]]` body
	// line on QA question notes. Same exclusion rationale as VocabBodyMarker.
	AnsweredByBodyMarker = "Answered by:"
	// AnswersBodyMarker prefixes the machine-written `Answers: [[…]]` body line on
	// QA answer notes. Same exclusion rationale as VocabBodyMarker.
	AnswersBodyMarker = "Answers:"
	// ContributorsBodyMarker prefixes the machine-written `Contributors: [[…]], …`
	// body line on QA answer notes. Excluded from BodyText/ContentHash so a
	// contributors-only write leaves the body vector and hash unchanged.
	ContributorsBodyMarker = "Contributors:"
	// RelatedSectionMarker is retained for backward compatibility: unmigrated
	// vault bodies still carry "Related to:" sections (the ritual was removed
	// 2026-07-02; bodies are stripped by the vocab migration). Hash exclusion
	// must keep working for them until that migration lands, then this can go.
	RelatedSectionMarker = "Related to:"
	// SupersedesBodyMarker prefixes the machine-written `Supersedes: [[…]] —
	// type: claim` body lines (replace-whole channel, written by learn/amend).
	// Excluded from BodyText/ContentHash for the same reason as VocabBodyMarker.
	// The cli writer's line matching aliases this constant — keep them in sync.
	SupersedesBodyMarker = "Supersedes:"
	// VocabBodyMarker prefixes the machine-written `Vocab: [[vocab.…]]` body
	// line (replace-whole channel, written by WriteVocabAssignment AFTER a
	// note is embedded). Excluding it from BodyText/ContentHash keeps a
	// vocab-assigning write from staling the sidecar and keeps [[vocab.…]]
	// wikilink noise out of body vectors on re-embed. The cli writer's line
	// matching aliases this constant — keep them in sync.
	VocabBodyMarker = "Vocab:"
)

Exported constants.

View Source 
const (
	BundledModelID = "minilm-l6-v2@384"
)

Exported constants.

View Source 
const (
	SidecarSchemaVersion = 1
)

Exported constants.

Variables

View Source 
var (
	ErrDimsMismatch     = errors.New("sidecar dims mismatch len(vector)")
	ErrSchemaVersion    = errors.New("sidecar schema version unsupported")
	ErrSidecarMalformed = errors.New("sidecar malformed")
)

Exported variables.

View Source 
var (
	ErrBundledModelUnavailable = errors.New(
		"bundled model missing or empty — rebuild the binary with the model in place, " +
			"or set ENGRAM_MODEL_PATH to a directory containing model.onnx",
	)
	ErrHugotEmbedEmpty = errors.New("hugot embed: empty result")
	ErrHugotProbeEmpty = errors.New("hugot probe returned no embedding")
)

Exported variables.

Functions

func BodyText 

func BodyText(raw []byte) []byte

BodyText returns the note body (frontmatter stripped) with all machine-written channel content removed: `Vocab:` and `Supersedes:` body lines (replace-whole channels) and any trailing "Related to:" section. It is the body-vector source for every note type. Dropping channel content means a channel-only edit (vocab assignment, supersession write, link edit) leaves the body vector and ContentHash unchanged (D3).

Machine lines are stripped BEFORE the Related-to pass: the writers append their lines after an unmigrated note's trailing "Related to:" block, and a non-bullet line after the block would otherwise disqualify it.

Trailing blank lines are normalized to a single newline LAST: the learn renderers end bodies with "\n\n" while the channel writers trim trailing blanks before appending their line — the original count is unrecoverable after a write, so the hash must be insensitive to it on both sides.

func ContentHash 

func ContentHash(raw []byte) string

ContentHash returns a sha256: prefixed hex digest covering BOTH embed sources — the situation: field and the body — joined by a 0x00 separator. Hashing both means staleness detection tracks either source: editing a note's situation OR its body changes the hash, marking the stored dual vectors stale.

func Cosine 

func Cosine(a, b []float32) float32

Cosine returns the cosine similarity of a and b. Returns 0 when either vector has zero magnitude or when lengths differ — callers should treat that as "no signal" rather than a strong match.

func ExtractBody 

func ExtractBody(raw []byte) []byte

ExtractBody returns the markdown body of a note with the leading YAML frontmatter block stripped. If the note has no leading frontmatter, it is returned unchanged.

Frontmatter format: a leading "---\n" line, arbitrary lines (which may themselves be blank), and a closing "---\n" line. Anything after the closing delimiter is the body. A single leading blank line after the closing delimiter is also stripped so notes whose frontmatter blocks differ but whose bodies match produce identical hashes.

func MarshalSidecar 

func MarshalSidecar(s Sidecar) []byte

MarshalSidecar encodes s as compact JSON. Vectors are large; pretty- printing them wastes disk and noises downstream diffs.

json.Marshal of a Sidecar (a struct of typed-string / int / []float32 / string fields, none of which implement MarshalJSON) cannot fail — the encoder only errors on cyclic data or custom marshaler failures. We swallow the error pointer to avoid the unreachable branch confusing coverage tools.

func SidecarPath 

func SidecarPath(notePath string) string

SidecarPath returns the .vec.json path sibling to a note's .md path. Non-.md inputs get .vec.json appended unchanged (defensive).

func SituationText 

func SituationText(raw []byte) []byte

SituationText returns the `situation:` frontmatter field for any note type ("" when absent or unparseable). It is the situation-vector source.

Types

type Embedder 

type Embedder interface {
	Embed(ctx context.Context, text string) ([]float32, error)
	ModelID() string
	Dims() int
}

Embedder produces fixed-dimension dense vectors from text. Implementations are expected to be safe for concurrent use unless documented otherwise.

type FS 

type FS interface {
	ReadFile(path string) ([]byte, error)
}

FS is the read-only filesystem surface used by ComputeState. The production reader returns *os.PathError which satisfies errors.Is for fs.ErrNotExist; test fakes can hand any error implementing IsNotExist() bool — the interface fallback covers them.

type HugotEmbedder 

type HugotEmbedder struct {
	// contains filtered or unexported fields
}

HugotEmbedder wraps a Hugot pipeline. Safe for concurrent use — Hugot's pipeline runs the model under its own lock.

func NewBundledHugotEmbedder 

func NewBundledHugotEmbedder(ctx context.Context, cacheDir string) (*HugotEmbedder, error)

NewBundledHugotEmbedder is the production constructor: bundled assets FS, fixed model directory, fixed model ID, and a caller-supplied cache dir. The cache dir is the XDG-keyed path where the model is extracted once and reused across all subsequent invocations.

func NewHugotEmbedderFromDir 

func NewHugotEmbedderFromDir(
	ctx context.Context,
	modelDir, modelID string,
) (*HugotEmbedder, error)

NewHugotEmbedderFromDir constructs an embedder reading the model from a directory on disk. Thin wrapper over buildEmbedder with the production Hugot backend; tests use buildEmbedder directly with a fake backend to exercise every error branch.

func NewHugotEmbedderFromFS 

func NewHugotEmbedderFromFS(
	ctx context.Context, modelFS stdembed.FS, modelDir, modelID, cacheDir string,
) (*HugotEmbedder, error)

NewHugotEmbedderFromFS constructs an embedder from any stdembed.FS rooted at modelDir. cacheDir is the stable directory where the model is extracted once and reused across invocations (XDG-keyed). Tests pass an empty FS to verify UAT 10's clear-error path; production wraps bundled assets.

func (*HugotEmbedder) Close 

func (h *HugotEmbedder) Close() error

Close releases the Hugot session. Safe to call multiple times. The model cache dir is NOT removed — it is a shared, persistent cache reused across all engram invocations.

func (*HugotEmbedder) Dims 

func (h *HugotEmbedder) Dims() int

Dims reports the embedding dimensionality.

func (*HugotEmbedder) Embed 

func (h *HugotEmbedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed runs the pipeline on text (truncated to fit the model's context window) and returns the resulting vector.

The char guard assumes prose density; code-dense text can still exceed the model's 512-token positional limit within the char limit (observed: 1500 chars of transcript tokenizing to 538 tokens, panicking graph compilation). On failure the input is halved and retried until it succeeds or bottoms out, so a single dense chunk degrades to a shorter prefix instead of failing the whole ingest.

func (*HugotEmbedder) ModelID 

func (h *HugotEmbedder) ModelID() string

ModelID reports the configured model identifier.

type LazyEmbedder 

type LazyEmbedder struct {
	// contains filtered or unexported fields
}

LazyEmbedder defers construction of an embedder until first use so commands that don't need it (help, update, transcript) don't pay the model-unpack cost or die if model loading fails. The construction is factory-injected so tests can drive both the success and failure init paths without running real Hugot.

func NewLazyEmbedder 

func NewLazyEmbedder(cacheDir string) *LazyEmbedder

NewLazyEmbedder returns a wrapper around NewBundledHugotEmbedder that extracts the bundled model to cacheDir at most once (on first Embed / ModelID / Dims call). cacheDir should be the XDG-keyed stable cache path for the model, e.g. $XDG_CACHE_HOME/engram/models/<model_id>/.

func (*LazyEmbedder) Dims 

func (l *LazyEmbedder) Dims() int

Dims lazily constructs the embedder, then delegates. Returns 0 when construction failed; callers should detect via an Embed error.

func (*LazyEmbedder) Embed 

func (l *LazyEmbedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed lazily constructs the embedder, then delegates.

func (*LazyEmbedder) ModelID 

func (l *LazyEmbedder) ModelID() string

ModelID lazily constructs the embedder, then delegates. Returns the bundled model id when construction has not been attempted yet so status-style callers can avoid paying the unpack cost.

type Sidecar 

type Sidecar struct {
	SchemaVersion    int       `json:"schema_version"`
	EmbeddingModelID string    `json:"embedding_model_id"`
	Dims             int       `json:"dims"`
	SituationVector  []float32 `json:"situation_vector"`
	BodyVector       []float32 `json:"body_vector"`
	ContentHash      string    `json:"content_hash"`
	// LastUsed is the date (YYYY-MM-DD) this note last surfaced as a useful
	// (above-cutoff) recall hit. Additive metadata: omitempty, EXCLUDED from
	// ContentHash (hash.go hashes situation+body of the raw note, not this), and
	// it does NOT bump SidecarSchemaVersion — old sidecars decode LastUsed=""
	// ("never used"). Never feed LastUsed into any hash: bumping it must not
	// mark a note stale.
	LastUsed string `json:"last_used,omitempty"` //nolint:tagliatelle // sidecar JSON keys are spec contract
}

Sidecar is the on-disk shape of a per-note .vec.json file. Field order here is the JSON key order. Snake-case keys match the spike spec's sidecar contract verbatim and are part of the on-disk file format: MiniLM-L6-v2@384 is the shipped bundled model, and the 2026-05-24 query spike froze the snake_case sidecar keys as a file format. Each note carries two vectors — one for its situation: frontmatter field and one for its body — so retrieval can match by max(situation, body).

func BuildSidecar 

func BuildSidecar(ctx context.Context, embedder Embedder, raw []byte) (Sidecar, error)

BuildSidecar embeds a note's situation and body and returns a fully stamped dual-vector sidecar. When the note has no situation field, the body text stands in for the situation embedding so every note still carries a meaningful situation vector. Either embed failure is returned to the caller, which applies its own warn-or-fail policy.

func UnmarshalSidecar 

func UnmarshalSidecar(data []byte) (Sidecar, error)

UnmarshalSidecar decodes a sidecar from JSON, returning ErrSidecarMalformed on parse failure, ErrSchemaVersion when the on-disk schema is not the current one (e.g. an old single-vector sidecar), or ErrDimsMismatch when either vector's length disagrees with Dims. The schema check precedes the vector-length check so an old sidecar (whose new vector fields decode empty) classifies as a schema mismatch rather than a dims mismatch.

type State 

type State int

State is the relationship between a note and its sidecar relative to the current binary's embedder. 

const (
	StateOK State = iota
	StateMissing
	StateStale
	StateIncompatible
	StateBroken
)

State values.

func ComputeState 

func ComputeState(filesystem FS, notePath, currentModelID string) State

ComputeState reads notePath and the sibling .vec.json and returns the note's State relative to currentModelID. Stale-vs-incompatible precedence: model_id mismatch first (a re-embed under the new model also picks up any body change), then content_hash mismatch.

ComputeState never returns an error: every failure mode classifies as a State (Missing for absent sidecar; Broken for unreadable note, unreadable sidecar, malformed JSON, or dims mismatch). The State IS the report so vault-wide passes can iterate without short-circuiting.

func (State) String 

func (s State) String() string

String returns the lowercase label for s used by `engram embed status`.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.