embedder

Documentation

Overview

Package embedder handles batch embedding of chunks and sparse vector computation.

Index

• Constants
• func EmbedAll(ctx context.Context, chunks []chunker.Chunk, ep provider.EmbeddingProvider, ...) ([][]float32, error)
• type SparseVector
  • func BM25Sparse(text string) SparseVector

Constants

const DefaultBatchSize = 96 // VoyageAI allows up to 128; 96 leaves headroom

Functions

func EmbedAll

func EmbedAll(
	ctx context.Context,
	chunks []chunker.Chunk,
	ep provider.EmbeddingProvider,
	batchSize int,
) ([][]float32, error)

EmbedAll embeds every chunk in batches and returns dense vectors in the same order as the input slice.

Types

type SparseVector

type SparseVector struct {
	Indices []uint32  `json:"indices"`
	Values  []float32 `json:"values"`
}

SparseVector is the Qdrant sparse vector format (indices + values, both same length).

func BM25Sparse

func BM25Sparse(text string) SparseVector

BM25Sparse computes a TF-based sparse vector for the text.

Algorithm:

1. Tokenise: lowercase + split on non-letter/non-digit runes, drop tokens < 2 chars.
2. Compute normalised term frequency (TF = count / total).
3. Map each token to an index via FNV-32a % maxSparseIndex (no global vocab needed).

This approximates sparse retrieval without requiring a corpus-wide IDF table. When combined with dense vectors via Qdrant RRF, it significantly improves recall on exact keyword matches (column names, codes, dates, etc.).