docsearch

Documentation

Overview

Package docsearch chunks local documents and indexes them with a deterministic, dependency-free vector embedding (feature-hashed character n-grams). Queries are embedded the same way and matched by cosine similarity, so only the most relevant fragments get surfaced to the agent. Everything runs locally — no ML models, no network, no external libraries.

Index

• func CacheKey(root string) string
• func Cosine(a, b map[int]float64) float64
• func Embed(text string) map[int]float64
• func MergeFetched(root, name, text string) (int, error)
• func ReembedFetch(root, name string, e Embedder) (int, error)
• func SetSemanticEmbedder(e Embedder)
• type Chunk
  • func ChunkText(text string, maxChars int) []Chunk
• type Doc
• type Embedder
  • func GetSemanticEmbedder() Embedder
• type Index
  • func IndexDir(root string) (*Index, error)
  • func IndexDirSemantic(root string, e Embedder) (*Index, error)
  • func Load(root string) *Index
  • func (ix *Index) Save() error
  • func (ix *Index) Search(query string, k int) []Score
• type Score

Functions

func CacheKey

func CacheKey(root string) string

CacheKey returns the persisted-index cache key for root.

func Cosine

func Cosine(a, b map[int]float64) float64

Cosine similarity between two embeddings.

func Embed

func Embed(text string) map[int]float64

Embed maps text to a fixed-dimension sparse vector using feature-hashed word and character n-grams. Deterministic: identical text always yields an identical vector, so a locally-embedded query matches locally-embedded documents exactly.

Three feature types are hashed into the same vector space:

• Whole words (exact keyword match boost)
• Word bigrams (phrase matching)
• Character 3-grams (fuzzy/morphological matching)

Without whole-word features, the cosine similarity between a query and a chunk that shares all the same words can be as low as 0.03 (the char 3-grams rarely overlap across different words). Whole-word features ensure that exact keyword matches produce a strong signal.

func MergeFetched

func MergeFetched(root, name, text string) (int, error)

MergeFetched merges an externally fetched document into root's persisted index under "fetch/<name>.md", replacing any prior version of that document and persisting the index. The text is chunked and embedded with the same deterministic n-gram vectors as on-disk docs, so it is searchable with Search. Returns the number of chunks added.

func ReembedFetch

func ReembedFetch(root, name string, e Embedder) (int, error)

ReembedFetch attaches dense embeddings (via a local Ollama embedder) to the fetched document "fetch/<name>.md" already in root's persisted index, then saves. Embedding failures for individual chunks are skipped, matching IndexDirSemantic. Returns the number of chunks embedded.

func SetSemanticEmbedder

func SetSemanticEmbedder(e Embedder)

SetSemanticEmbedder atomically replaces the package-level SemanticEmbedder.

Types

type Chunk

type Chunk struct {
	File  string `json:"file"`
	Start int    `json:"start"` // 1-based first line of the chunk
	Text  string `json:"text"`
}

Chunk is one contiguous fragment of a source document.

func ChunkText

func ChunkText(text string, maxChars int) []Chunk

ChunkText splits text into contiguous chunks of at most maxChars, keeping paragraphs together where possible. Chunks shorter than 40 chars are dropped (they carry no retrieval value). Start is the 1-based line number.

type Doc

type Doc struct {
	ID    string          `json:"id"`
	Chunk Chunk           `json:"chunk"`
	Vec   map[int]float64 `json:"vec"`
	// Semantic is an optional dense embedding (e.g. from a local Ollama
	// model) added by IndexDirSemantic. When present it is fused into Search
	// alongside the deterministic hashed Vec and BM25.
	Semantic []float32 `json:"semantic,omitempty"`
}

Doc is an indexed chunk with its embedding.

type Embedder

type Embedder interface {
	EmbedText(text string) ([]float32, error)
}

Embedder produces dense semantic embeddings for text. internal/llm.Client implements it against a local Ollama server. It is optional: docsearch's deterministic n-gram embedding remains the always-available fallback.

var (
	SemanticEmbedder Embedder
)

SemanticEmbedder, when non-nil, embeds search queries for indexes that carry dense Doc.Semantic vectors. It must be the same embedder used to build the index (the CLI/MCP layer sets it when it indexed the docs).

It is written and read from concurrent handler goroutines, so all access is serialized through semanticEmbedderMu and the Get/Set accessors below. Do not read or write the field directly.

func GetSemanticEmbedder

func GetSemanticEmbedder() Embedder

GetSemanticEmbedder returns the current SemanticEmbedder (nil when unset), guarding the read against concurrent writes.

type Index

type Index struct {
	Root string `json:"root"`
	Docs []Doc  `json:"docs"`
	// contains filtered or unexported fields
}

Index is a set of embedded chunks for one root.

func IndexDir

func IndexDir(root string) (*Index, error)

IndexDir walks root and chunks + embeds every document file. It returns the in-memory index (callers may persist it with Save).

func IndexDirSemantic

func IndexDirSemantic(root string, e Embedder) (*Index, error)

IndexDirSemantic is IndexDir plus an optional dense embedding pass: every chunk is also embedded through e and stored in Doc.Semantic. Embedding failures for individual chunks are skipped (the deterministic Vec is always present), so a partially-available model still yields a usable index.

func Load

func Load(root string) *Index

Load reads a previously persisted index for root. Returns nil when absent.

func (*Index) Save

func (ix *Index) Save() error

Save persists the index to the local cache.

func (*Index) Search

func (ix *Index) Search(query string, k int) []Score

Search returns the top-k chunks most relevant to query, best first. Three signals are fused by reciprocal rank: the cosine similarity of the feature-hashed n-gram embeddings (fuzzy, morphological), the cosine similarity of the optional dense semantic embeddings (real meaning, when the index was built with IndexDirSemantic), and BM25 over the chunk's words (exact keyword matching). A chunk matching no signal is omitted. Sim holds the fused RRF score.

type Score

type Score struct {
	Doc Doc
	Sim float64
}

Score ranks a chunk against a query.