index

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 21, 2026 License: MIT Imports: 13 Imported by: 0

README

index

Storage indexes: in-memory vector + keyword indexes, ANN search, metadata filtering, and quantized variants. This is the retrieval engine under store.

Indexes

Index Constructor Notes
MemoryIndex NewMemoryIndex(ns, dim) Brute-force cosine similarity + internal BM25; the workhorse.
HNSW NewHNSW(dim, cfg) Hierarchical Navigable Small World ANN graph; auto-enabled by stores at 1K+ chunks (HNSWThreshold).
HybridIndex NewHybridIndex(ns, dim, fusion) Vector + BM25 with a pluggable fuse.Fusion.
MetadataIndex NewMetadataIndex() Pure metadata filtering.
MultiVectorIndex NewMultiVectorIndex(ns, dim) Multiple vectors per item with aggregation modes.
QuantizedIndex / PQIndex NewQuantizedIndex / NewPQIndex Scalar / product quantization for memory reduction.

Search options & filters

SearchOptions (TopK, MinScore, Filters, Hybrid, BM25Weight, Fusion, EfSearch) parameterizes every search. Filters combine conjunctively:

  • TermFilter — metadata term equality
  • RangeFilter — numeric range
  • DateRangeFilter — time range

SortedIDs renders a set of IDs deterministically (used in tests/goldens).

Documentation

Overview

Package index provides the interface and implementations for storing and retrieving text chunks with their embeddings.

Package index provides the interface and implementations for storing and retrieving text chunks with their embeddings.

Index

Constants

View Source 
const HNSWThreshold = 1000

HNSWThreshold is the default number of chunks above which HNSW is used for search. Per-index overrides are available via MemoryIndex.SetHNSWThreshold.

Variables

This section is empty.

Functions

func SortedIDs 

func SortedIDs(set map[string]struct{}) []string

SortedIDs converts an ID set to a sorted slice for deterministic iteration.

Types

type Aggregation 

type Aggregation int

Aggregation selects how a chunk's multiple embeddings are collapsed into a single similarity score. 

const (
	// MaxSimAggregation scores a chunk by its best (max) single-vector
	// similarity — the ColBERT-style MaxSim upper bound. Default.
	MaxSimAggregation Aggregation = iota

	// MeanAggregation scores a chunk by the mean of all its vector
	// similarities. Robust to one spurious vector.
	MeanAggregation

	// TopMeanAggregation scores a chunk by the mean of its TopN best
	// similarities (0 falls back to MaxSim semantics when TopN == 1).
	TopMeanAggregation
)

type DateRangeFilter 

type DateRangeFilter struct {
	Key     string // metadata key containing a time string
	Min     *time.Time
	Max     *time.Time
	MinIncl bool
	MaxIncl bool
}

DateRangeFilter matches chunks where the CreatedAt is in a date range.

func (*DateRangeFilter) Match 

func (f *DateRangeFilter) Match(chunk *core.Chunk) bool

type Filter 

type Filter interface {
	// Match returns true if the chunk's metadata matches this filter.
	Match(chunk *core.Chunk) bool
}

Filter is a metadata filter for search results.

type HNSW 

type HNSW struct {
	// contains filtered or unexported fields
}

func NewHNSW 

func NewHNSW(dim int, cfg HNSWConfig) *HNSW

func (*HNSW) Add 

func (h *HNSW) Add(id string, embedding []float32)

Add inserts a new node into the HNSW graph.

func (*HNSW) Contains 

func (h *HNSW) Contains(id string) bool

Contains reports whether a node with the given ID is present in the graph.

func (*HNSW) Search 

func (h *HNSW) Search(query []float32, ef int) []string

Search finds the nearest neighbors to the query embedding.

type HNSWConfig 

type HNSWConfig struct {
	M              int
	M0             int
	EfConstruction int
	EfSearch       int
}

HNSWConfig holds configuration for the HNSW index.

func DefaultHNSWConfig 

func DefaultHNSWConfig() HNSWConfig

DefaultHNSWConfig returns standard HNSW parameters.

type HybridIndex 

type HybridIndex struct {
	// contains filtered or unexported fields
}

HybridIndex combines a dense vector index and a sparse BM25 keyword index over the same chunks into a single index that serves fused hybrid search. Vector-only or keyword-only hits are both retained: a chunk that ranks only on keywords (e.g. exact technical terms with weak semantic overlap) still surfaces.

func NewHybridIndex 

func NewHybridIndex(ns string, dim int, fusion fuse.Fusion) *HybridIndex

NewHybridIndex creates a HybridIndex. A nil fusion means Search will weight the two score maps by SearchOptions.BM25Weight (0.5/0.5 when unset).

func (*HybridIndex) Add 

func (h *HybridIndex) Add(ctx context.Context, chunk *core.Chunk) error

Add inserts a chunk into both sub-indexes.

func (*HybridIndex) Count 

func (h *HybridIndex) Count() int

Count returns the number of indexed chunks.

func (*HybridIndex) Delete 

func (h *HybridIndex) Delete(ctx context.Context, id string) error

Delete removes a chunk from both sub-indexes.

func (*HybridIndex) Dimension 

func (h *HybridIndex) Dimension() int

Dimension returns the embedding dimension.

func (*HybridIndex) GetChunk 

func (h *HybridIndex) GetChunk(id string) (*core.Chunk, bool)

GetChunk returns a chunk by ID.

func (*HybridIndex) Namespace 

func (h *HybridIndex) Namespace() string

Namespace returns the namespace of this index.

func (*HybridIndex) Search 

func (h *HybridIndex) Search(ctx context.Context, query string, queryEmb []float32, opts SearchOptions) ([]SearchResult, error)

Search performs fused hybrid search: dense similarity against queryEmb plus BM25 ranking against the raw query text.

func (*HybridIndex) SearchBM25 

func (h *HybridIndex) SearchBM25(query string) []bm25.SearchResult

SearchBM25 exposes the keyword sub-index directly.

type Index 

type Index interface {
	// Add inserts a chunk into the index. The chunk's embedding must be non-nil.
	Add(ctx context.Context, chunk *core.Chunk) error

	// AddBatch inserts multiple chunks into the index.
	AddBatch(ctx context.Context, chunks []*core.Chunk) error

	// Delete removes a chunk from the index by its ID.
	Delete(ctx context.Context, id string) error

	// Search finds the most similar chunks to the given query embedding.
	Search(ctx context.Context, query []float32, opts SearchOptions) ([]SearchResult, error)

	// Count returns the number of chunks in the index.
	Count() int

	// Dimension returns the embedding dimension of the index.
	Dimension() int

	// Namespace returns the namespace of this index.
	Namespace() string
}

Index defines the interface for storing and retrieving embedded chunks.

type MemoryIndex 

type MemoryIndex struct {
	// contains filtered or unexported fields
}

MemoryIndex is an in-memory index that stores chunks and their embeddings. For datasets > HNSWThreshold, it uses an HNSW graph for approximate nearest neighbor search.

func NewMemoryIndex 

func NewMemoryIndex(namespace string, dimension int) *MemoryIndex

NewMemoryIndex creates a new in-memory index.

func (*MemoryIndex) Add 

func (m *MemoryIndex) Add(_ context.Context, chunk *core.Chunk) error

Add inserts a chunk into the index.

func (*MemoryIndex) AddBatch 

func (m *MemoryIndex) AddBatch(_ context.Context, chunks []*core.Chunk) error

AddBatch inserts multiple chunks into the index.

func (*MemoryIndex) Count 

func (m *MemoryIndex) Count() int

Count returns the number of chunks in the index.

func (*MemoryIndex) Delete 

func (m *MemoryIndex) Delete(_ context.Context, id string) error

Delete removes a chunk from the index. It returns core.ErrNotFound when the ID is not present.

func (*MemoryIndex) Dimension 

func (m *MemoryIndex) Dimension() int

Dimension returns the embedding dimension.

func (*MemoryIndex) GetChunk 

func (m *MemoryIndex) GetChunk(id string) (*core.Chunk, bool)

GetChunk returns a chunk by ID.

func (*MemoryIndex) Namespace 

func (m *MemoryIndex) Namespace() string

Namespace returns the namespace.

func (*MemoryIndex) Search 

func (m *MemoryIndex) Search(_ context.Context, query []float32, opts SearchOptions) ([]SearchResult, error)

Search finds the most similar chunks to the given query embedding.

func (*MemoryIndex) SearchBM25 

func (m *MemoryIndex) SearchBM25(query string) []bm25.SearchResult

SearchBM25 returns keyword (BM25) matches for the query from this index's internal BM25 index, sorted by score descending. This is the index's single source of keyword state: documents are added on Add/AddBatch and pruned on Delete, so deleted chunks never match.

func (*MemoryIndex) SetHNSWThreshold 

func (m *MemoryIndex) SetHNSWThreshold(n int)

SetHNSWThreshold overrides the chunk count above which HNSW search activates (default HNSWThreshold). It only takes effect before the HNSW graph has been built; a non-positive value restores the default.

type MetadataIndex 

type MetadataIndex struct {
	// contains filtered or unexported fields
}

MetadataIndex is an inverted index over chunk metadata that answers "which chunks could match these filters?" in set operations instead of scanning every chunk. It complements vector indexes: query the metadata index first to get a candidate set, then restrict the vector search to it (or use Candidates directly for pure metadata lookups).

func NewMetadataIndex 

func NewMetadataIndex() *MetadataIndex

NewMetadataIndex creates an empty MetadataIndex.

func (*MetadataIndex) Add 

func (mi *MetadataIndex) Add(chunk *core.Chunk) error

Add indexes a chunk's metadata. Re-adding an existing ID replaces the previous entry.

func (*MetadataIndex) Candidates 

func (mi *MetadataIndex) Candidates(filters []Filter) (map[string]struct{}, bool)

Candidates evaluates the given filters and returns the set of chunk IDs that could match, or (nil, false) when there are no filters (meaning "no pre-filter needed").

TermFilter and TermInFilter are answered with posting-list intersections/unions; any other Filter is evaluated by scanning the current candidate set with Filter.Match.

func (*MetadataIndex) Count 

func (mi *MetadataIndex) Count() int

Count returns the number of indexed chunks.

func (*MetadataIndex) Get 

func (mi *MetadataIndex) Get(id string) (*core.Chunk, bool)

Get returns a stored chunk by ID.

func (*MetadataIndex) Remove 

func (mi *MetadataIndex) Remove(id string) error

Remove deletes a chunk and its postings.

func (*MetadataIndex) Values 

func (mi *MetadataIndex) Values(key string) []string

Values lists the indexed metadata values for a key, sorted.

type MultiVectorIndex 

type MultiVectorIndex struct {

	// Aggregation selects the scoring mode. Zero value = MaxSim.
	Aggregation Aggregation

	// TopN is the number of best vectors used by TopMeanAggregation.
	TopN int
	// contains filtered or unexported fields
}

MultiVectorIndex stores multiple embeddings per chunk (e.g. one per passage segment, or separate "query-side" and "passage-side" vectors) and scores each stored vector against the query, aggregating per the configured Aggregation. This supports retrieval models where a single vector under-represents a chunk.

func NewMultiVectorIndex 

func NewMultiVectorIndex(ns string, dim int) *MultiVectorIndex

NewMultiVectorIndex creates a MultiVectorIndex for the given namespace and embedding dimension.

func (*MultiVectorIndex) Add 

func (m *MultiVectorIndex) Add(_ context.Context, chunk *core.Chunk) error

Add indexes a chunk using its single Embedding as the one-vector multi-set.

func (*MultiVectorIndex) AddBatch 

func (m *MultiVectorIndex) AddBatch(ctx context.Context, chunks []*core.Chunk) error

AddBatch indexes multiple single-embedding chunks.

func (*MultiVectorIndex) AddMulti 

func (m *MultiVectorIndex) AddMulti(_ context.Context, chunk *core.Chunk, vectors [][]float32) error

AddMulti indexes a chunk with an explicit set of embeddings. Each vector must have the index dimension and the set must be non-empty.

func (*MultiVectorIndex) Count 

func (m *MultiVectorIndex) Count() int

Count returns the number of chunks in the index.

func (*MultiVectorIndex) Delete 

func (m *MultiVectorIndex) Delete(_ context.Context, id string) error

Delete removes a chunk and all of its vectors.

func (*MultiVectorIndex) Dimension 

func (m *MultiVectorIndex) Dimension() int

Dimension returns the embedding dimension.

func (*MultiVectorIndex) GetChunk 

func (m *MultiVectorIndex) GetChunk(id string) (*core.Chunk, bool)

GetChunk returns a chunk by ID.

func (*MultiVectorIndex) Namespace 

func (m *MultiVectorIndex) Namespace() string

Namespace returns the namespace of this index.

func (*MultiVectorIndex) Search 

func (m *MultiVectorIndex) Search(_ context.Context, query []float32, opts SearchOptions) ([]SearchResult, error)

Search scores every stored vector against the query and aggregates per chunk.

func (*MultiVectorIndex) VectorCount 

func (m *MultiVectorIndex) VectorCount(id string) (int, bool)

VectorCount returns how many embeddings are stored for a chunk.

type PQIndex 

type PQIndex struct {
	// contains filtered or unexported fields
}

PQIndex is a memory index that stores product-quantized codes instead of raw float32 vectors. Vectors are L2-normalized before encoding so ADC distances correspond to cosine similarity: for unit vectors, cos(a, b) = 1 - ||a-b||^2 / 2.

The index must be created with an already-trained ProductQuantizer.

func NewPQIndex 

func NewPQIndex(ns string, pq *ProductQuantizer) (*PQIndex, error)

NewPQIndex creates a PQIndex over the given trained quantizer.

func (*PQIndex) Add 

func (p *PQIndex) Add(_ context.Context, chunk *core.Chunk) error

Add inserts a chunk. The chunk must carry a non-empty embedding; the quantizer must be trained.

func (*PQIndex) AddBatch 

func (p *PQIndex) AddBatch(ctx context.Context, chunks []*core.Chunk) error

AddBatch inserts multiple chunks.

func (*PQIndex) Count 

func (p *PQIndex) Count() int

Count returns the number of indexed chunks.

func (*PQIndex) Delete 

func (p *PQIndex) Delete(_ context.Context, id string) error

Delete removes a chunk from the index.

func (*PQIndex) Dimension 

func (p *PQIndex) Dimension() int

Dimension returns the vector dimension.

func (*PQIndex) MemoryBytes 

func (p *PQIndex) MemoryBytes() int64

MemoryBytes reports how many bytes the quantized codes occupy.

func (*PQIndex) Namespace 

func (p *PQIndex) Namespace() string

Namespace returns the namespace of this index.

func (*PQIndex) Search 

func (p *PQIndex) Search(_ context.Context, query []float32, opts SearchOptions) ([]SearchResult, error)

Search scores stored codes against the query with addable quantization: one distance table is built per query, then each code is scored in O(m).

type ProductQuantizer 

type ProductQuantizer struct {
	// contains filtered or unexported fields
}

ProductQuantizer implements product quantization (PQ): a vector of dimension dim is split into m contiguous sub-vectors of dimension dim/m, and each sub-vector is encoded as the index of its nearest centroid in a per-subspace codebook of k centroids (fitted with k-means++). A full vector therefore compresses from dim*4 bytes to m bytes (e.g. 768-dim with m=96 -> 96 bytes, an 8x reduction), and distances at search time are computed with addable quantization (ADC) using precomputed distance tables.

func NewProductQuantizer 

func NewProductQuantizer(dim, m, k int) (*ProductQuantizer, error)

NewProductQuantizer creates a PQ configuration. dim must be divisible by m, and k must be at least 2.

func (*ProductQuantizer) Decode 

func (p *ProductQuantizer) Decode(code []uint8) ([]float32, error)

Decode reconstructs the full-length vector from a code by concatenating the chosen centroids.

func (*ProductQuantizer) Dimensions 

func (p *ProductQuantizer) Dimensions() (dim, m int)

Dimensions reports the configured dim and m (subspaces).

func (*ProductQuantizer) DistanceTable 

func (p *ProductQuantizer) DistanceTable(query []float32) ([][]float64, error)

DistanceTable precomputes, for a query vector, the squared L2 distance from each query sub-vector to every centroid of the matching subspace. Search then scores a stored code by summing one table entry per subspace — O(m) instead of O(dim).

func (*ProductQuantizer) Encode 

func (p *ProductQuantizer) Encode(vec []float32) ([]uint8, error)

Encode maps a vector to m code bytes (one per subspace).

func (*ProductQuantizer) Train 

func (p *ProductQuantizer) Train(vectors [][]float32) error

Train fits per-subspace codebooks via k-means++ over the provided vectors.

func (*ProductQuantizer) Trained 

func (p *ProductQuantizer) Trained() bool

Trained reports whether codebooks are fitted.

func (*ProductQuantizer) WithSeed 

func (p *ProductQuantizer) WithSeed(seed int64) *ProductQuantizer

WithSeed sets the RNG seed used for k-means++ initialization, making training deterministic for a given seed.

type QuantizedIndex 

type QuantizedIndex struct {
	// contains filtered or unexported fields
}

QuantizedIndex is a memory index that stores 8-bit scalar-quantized vectors instead of raw float32s, cutting vector memory 4x. Search dequantizes candidate vectors, so results are exact with respect to the quantized representation.

The index must be created with an already-trained ScalarQuantizer.

func NewQuantizedIndex 

func NewQuantizedIndex(ns string, qz *ScalarQuantizer) (*QuantizedIndex, error)

NewQuantizedIndex creates a QuantizedIndex over the given trained quantizer.

func (*QuantizedIndex) Add 

func (q *QuantizedIndex) Add(_ context.Context, chunk *core.Chunk) error

Add inserts a chunk. The chunk must carry an embedding of the correct dimension; the quantizer must be trained.

func (*QuantizedIndex) AddBatch 

func (q *QuantizedIndex) AddBatch(ctx context.Context, chunks []*core.Chunk) error

AddBatch inserts multiple chunks.

func (*QuantizedIndex) Count 

func (q *QuantizedIndex) Count() int

Count returns the number of indexed chunks.

func (*QuantizedIndex) Delete 

func (q *QuantizedIndex) Delete(_ context.Context, id string) error

Delete removes a chunk from the index.

func (*QuantizedIndex) Dimension 

func (q *QuantizedIndex) Dimension() int

Dimension returns the vector dimension.

func (*QuantizedIndex) MemoryBytes 

func (q *QuantizedIndex) MemoryBytes() int64

MemoryBytes reports how many bytes the quantized vectors occupy.

func (*QuantizedIndex) Namespace 

func (q *QuantizedIndex) Namespace() string

Namespace returns the namespace of this index.

func (*QuantizedIndex) Search 

func (q *QuantizedIndex) Search(_ context.Context, query []float32, opts SearchOptions) ([]SearchResult, error)

Search finds the most similar chunks to the query vector using dequantized candidates.

type RangeFilter 

type RangeFilter struct {
	Key     string
	Min     *float64
	Max     *float64
	MinIncl bool
	MaxIncl bool
}

RangeFilter matches chunks where a numeric metadata value is in a range.

func (*RangeFilter) Match 

func (f *RangeFilter) Match(chunk *core.Chunk) bool

type ScalarQuantizer 

type ScalarQuantizer struct {
	// contains filtered or unexported fields
}

ScalarQuantizer implements 8-bit scalar quantization (SQ8): each embedding dimension is mapped to a single byte using a per-dimension linear scale fitted during Train. This compresses vectors 4x (dim*4 bytes -> dim bytes) at a small, bounded recall cost.

func NewScalarQuantizer 

func NewScalarQuantizer(dim int) (*ScalarQuantizer, error)

NewScalarQuantizer creates a quantizer for vectors of the given dimension. It must be trained (Train) before Quantize is called.

func (*ScalarQuantizer) Dequantize 

func (q *ScalarQuantizer) Dequantize(code []uint8) ([]float32, error)

Dequantize maps a byte code back to a float vector.

func (*ScalarQuantizer) Dimension 

func (q *ScalarQuantizer) Dimension() int

Dimension returns the vector dimension this quantizer was built for.

func (*ScalarQuantizer) MeanAbsError 

func (q *ScalarQuantizer) MeanAbsError(vectors [][]float32) (float64, error)

MeanAbsError measures the average per-dimension reconstruction error over the given vectors. It is a practical measure of quantization loss.

func (*ScalarQuantizer) Quantize 

func (q *ScalarQuantizer) Quantize(vec []float32) ([]uint8, error)

Quantize maps a float vector to one byte per dimension.

func (*ScalarQuantizer) Train 

func (q *ScalarQuantizer) Train(vectors [][]float32) error

Train fits per-dimension min/max scales from the provided vectors. Training is idempotent: calling it again refits the scales.

func (*ScalarQuantizer) Trained 

func (q *ScalarQuantizer) Trained() bool

Trained reports whether the quantizer has fitted scales.

type SearchOptions 

type SearchOptions struct {
	// TopK is the maximum number of results to return.
	TopK int

	// Filters are metadata filters to apply before searching.
	Filters []Filter

	// MinScore is the minimum relevance score for a result to be included.
	// In hybrid mode it applies to the fused score, not the raw vector or
	// keyword scores.
	MinScore float64

	// Hybrid enables BM25 keyword search combined with vector similarity.
	Hybrid bool

	// BM25Weight is the weight for BM25 scores in hybrid search (0-1).
	// 0 = pure vector, 1 = pure BM25, 0.5 = equal weighting.
	BM25Weight float64

	// Fusion allows custom score fusion (overrides BM25Weight if set).
	Fusion fuse.Fusion

	// EfSearch controls the HNSW search width (only used when HNSW is active).
	// 0 means use the default (50).
	EfSearch int
}

SearchOptions configures a search operation.

func DefaultSearchOptions 

func DefaultSearchOptions(topK int) SearchOptions

DefaultSearchOptions returns SearchOptions with sensible defaults.

type SearchResult 

type SearchResult struct {
	// Chunk is the matching chunk.
	Chunk *core.Chunk

	// Score is the relevance score (higher is more similar).
	Score float64

	// RerankScore is the fine-rank score assigned by a reranker (higher is
	// more relevant). It is zero until a reranker has processed this result.
	RerankScore float64

	// RerankRank is the 1-based position this result occupies after
	// reranking. It is zero when no reranker has run.
	RerankRank int

	// Reranker is the name of the reranker that produced RerankScore, for
	// score attribution. Empty when no reranker has run.
	Reranker string
}

SearchResult represents a single result from a similarity search.

type TermFilter 

type TermFilter struct {
	Key   string
	Value string
}

TermFilter matches chunks where a metadata key equals a specific string value.

func (*TermFilter) Match 

func (f *TermFilter) Match(chunk *core.Chunk) bool

type TermInFilter 

type TermInFilter struct {
	Key    string
	Values []string
}

TermInFilter matches chunks where a metadata key's value is in a set.

func (*TermInFilter) Match 

func (f *TermInFilter) Match(chunk *core.Chunk) bool

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.