bm25

README

bm25

A pure-Go Okapi BM25 keyword ranking function used for the keyword leg of hybrid search. No external dependencies.

API

bm := bm25.New(bm25.DefaultConfig())
bm.AddDocument(chunkID, content)
results := bm.Search("query text", topK) // []SearchResult{ID, Score}
bm.RemoveDocument(chunkID)

Config exposes the standard BM25 parameters (K1, B, tokenization case-folding).

Used by

index.MemoryIndex / index.HybridIndex maintain a BM25 index alongside the vector index; store.SearchHybrid fuses the two score lists (see fuse).

Documentation

Overview

Package bm25 implements the BM25 ranking function for keyword search.

Index

• type BM25
  • func New(cfg Config) *BM25
  • func (b *BM25) AddDocument(docID string, content string) []string
  • func (b *BM25) Count() int
  • func (b *BM25) RemoveDocument(docID string)
  • func (b *BM25) Search(query string) []SearchResult
• type Config
  • func DefaultConfig() Config
• type SearchResult

Types

type BM25

type BM25 struct {
	// contains filtered or unexported fields
}

BM25 implements the BM25 ranking function.

func New

func New(cfg Config) *BM25

New creates a new BM25 index.

func (*BM25) AddDocument

func (b *BM25) AddDocument(docID string, content string) []string

AddDocument adds a document to the BM25 index. Returns tokenized tokens. Re-adding an existing docID replaces the previous version: its old postings are removed first so docCount, docFreq, and avgDocLen stay consistent.

func (*BM25) Count

func (b *BM25) Count() int

Count returns the number of indexed documents.

func (*BM25) RemoveDocument

func (b *BM25) RemoveDocument(docID string)

RemoveDocument removes a document from the index. Removing an unknown docID is a no-op.

func (*BM25) Search

func (b *BM25) Search(query string) []SearchResult

Search returns BM25 scores sorted descending.

type Config

type Config struct {
	K1 float64 // Term saturation (default: 1.2)
	B  float64 // Length normalization (default: 0.75)
}

Config holds BM25 parameters.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns standard BM25 parameters.

type SearchResult

type SearchResult struct {
	DocID string
	Score float64
}

SearchResult represents a single BM25 result.