embedding

Documentation

Overview

Package embedding is the shared text-embedding seam used by every semantic retrieval path in odek: memory (episode recall, dedup, ranking, fact merge), session search, and skill matching. It abstracts over two embedding families behind a single TextEmbedder interface:

• corpus-fitted (RandomProjections): a fast, local, zero-dependency lexical backend over bag-of-words + bigrams.
• stateless (HTTP APIs): any OpenAI-compatible embeddings endpoint, giving real semantic similarity.

Extracted from internal/memory so sessions and skills can reuse the exact same backends, config shape, and fingerprint semantics.

Index

• Constants
• func Cosine(a, b vector.Vector) float32
• type Config
• type TextEmbedder
  • func New(cfg *Config, rpDims int) TextEmbedder
  • func NewRP(dims int) TextEmbedder

Constants

const DefaultRPDim = 256

DefaultRPDim is the default RandomProjections dimensionality used when a caller passes rpDims <= 0.

const DefaultTimeout = 10 * time.Second

DefaultTimeout bounds embedding API calls. Recall runs on the per-turn hot path, so a hung embedding server must never stall the loop for the HTTPEmbedder's default 30s.

Functions

func Cosine

func Cosine(a, b vector.Vector) float32

Cosine computes cosine similarity between two go-vector Vectors. Mismatched or empty inputs score 0. A buggy/hostile embedding backend can return NaN/Inf components; a NaN score breaks sort ordering (non-strict-weak), so it is clamped to 0 ("no similarity") to keep ranking well-defined.

Types

type Config

type Config struct {
	Provider string `json:"provider,omitempty"`
	// BaseURL is the API root, e.g. "http://localhost:11434/v1" (Ollama) or
	// "https://api.openai.com/v1"; "/embeddings" is appended by the client.
	BaseURL string `json:"base_url,omitempty"`
	// Model is the embedding model name, e.g. "nomic-embed-text" or
	// "text-embedding-3-small".
	Model string `json:"model,omitempty"`
	// Dims declares the expected vector dimensionality. 0 = infer from the
	// first response (recommended).
	Dims int `json:"dims,omitempty"`
	// APIKey is sent as "Authorization: Bearer <key>" when non-empty.
	APIKey string `json:"api_key,omitempty"`
	// TimeoutSeconds is the per-request HTTP timeout. Default: 10.
	TimeoutSeconds int `json:"timeout_seconds,omitempty"`
}

Config selects the embedding backend.

Provider values:

• "" or "rp" (default) — go-vector RandomProjections over bag-of-words + bigrams. Fast, local, zero-dependency, but purely lexical: texts with no shared vocabulary score 0 even when they mean the same thing.

• "http" — any OpenAI-compatible embeddings API (Ollama, llama.cpp server, LM Studio, vLLM, OpenAI, Voyage…) via go-vector's HTTPEmbedder. Real semantic similarity: "fixed the auth bug" matches "repaired login issue". Requires BaseURL and Model; if either is missing the config silently falls back to "rp" so retrieval keeps working.

BaseURL and APIKey support ${ENV_VAR} expansion so secrets can stay out of config files (e.g. "api_key": "${OPENAI_API_KEY}").

type TextEmbedder

type TextEmbedder interface {
	// Fit prepares the embedder for a corpus of raw (unfeaturized) texts.
	Fit(corpus []string) error
	// Embed returns the vector for one raw text.
	Embed(text string) (vector.Vector, error)
	// EmbedAll returns vectors for texts in order. A nil vector marks a
	// per-text failure only for implementations that embed one-by-one;
	// batch implementations fail atomically.
	EmbedAll(texts []string) ([]vector.Vector, error)
	// Fingerprint identifies the embedding space for persistence compat.
	Fingerprint() string
	// SaveState persists fitted state to path (RP gob). No-op when stateless.
	SaveState(path string)
	// LoadState restores fitted state from path. Returns true when the
	// embedder is usable afterwards (always true for stateless backends).
	LoadState(path string) bool
}

TextEmbedder is the seam between retrieval paths and the vector backend. It abstracts over the two embedding families:

• corpus-fitted (RandomProjections): Fit must run on the FULL corpus before Embed, and refit after the corpus changes, or new-term vectors are degenerate. State is per-instance — consumers that fit different corpora need separate instances.

• stateless (HTTP APIs): Fit only warms a cache; Embed works at any time and vectors are stable across corpus changes.

Fingerprint identifies the embedding space (provider/model/dims). Persisted vectors are only reusable by an embedder with the same fingerprint.

func New

func New(cfg *Config, rpDims int) TextEmbedder

New builds the embedder selected by cfg. rpDims sets the RandomProjections dimensionality used when cfg selects (or falls back to) the "rp" provider — callers pass their legacy dims so persisted RP state stays loadable. Invalid/incomplete "http" config falls back to "rp".

func NewRP

func NewRP(dims int) TextEmbedder

NewRP builds a RandomProjections embedder of the given dimensionality. Pass dims <= 0 for DefaultRPDim.