bm25

README

bm25

The sparse half of Qdrant hybrid search, in Go — byte-for-byte identical to Python FastEmbed, and a lot faster.

bm25 is a Go port of FastEmbed's Qdrant/bm25 sparse encoder. It produces the same sparse vectors as the Python encoder, so a Go service can run BM25 and hybrid search against a corpus indexed with FastEmbed — no Python, no sidecar, no inference server.

It's the missing piece for hybrid search in Go: the dense fastembed-go port doesn't do sparse models, and Qdrant's BM25 token ids are hashes of stemmed tokens — a query only matches the stored corpus if it's tokenized, stemmed, and hashed identically. This is, tested against output captured from FastEmbed itself.

Highlights

• Exact FastEmbed parity — verified byte-for-byte across fastembed 0.4.0–0.8.0, with a CI matrix that catches drift in new releases.
• Fast — ~3.5× faster per core, ~11.5× with parallel batch, and ~500× faster cold start than Python (numbers).
• 16 languages — each validated against FastEmbed's stemmer and stopwords.
• Hybrid-ready — drops straight into Qdrant dense+sparse RRF; the qhybrid module wraps the whole flow.
• Built for scale — concurrency-safe, multicore EncodeBatchParallel, streaming EncodeStream, allocation-light hot paths.
• Tiny footprint — pure Go, ~3 MB static binary, no model download, no ONNX Runtime; even compiles to WebAssembly.

Install

go get github.com/harsh04/bm25

Usage

enc := bm25.New()
indices, values := enc.Encode("running runs ran")
// indices: [243905464 567658162]
// values:  [1.9043111 1.6786885]

Two tokens out of three words: running and runs both stem to run (one id, weighted higher for the repeat), ran stems to ran. Stopwords drop out entirely — Encode("What is the cancellation policy?") keeps only cancel and polici. This is the same tokenize → stopword → stem → hash → BM25 pipeline as Python FastEmbed, producing the same uint32 token ids, so the vectors line up with a corpus FastEmbed indexed.

Values are term frequencies only — Qdrant applies IDF server-side via the sparse vector's modifier, so don't apply it yourself. Empty or stopword-only text returns empty slices, and an Encoder is safe to share across goroutines.

Hand the indices/values to the Qdrant Go client's NewVectorSparse / NewQuerySparse for the sparse arm of a query. There's a full runnable dense+sparse+RRF example in _examples/qdrant_hybrid, or use the qhybrid module for collection setup, upsert, and RRF search in a few lines.

Local search (no Qdrant)

Need BM25 ranking without standing up Qdrant — tests, CLIs, edge, small corpora? Index does full BM25 (with IDF) in memory, with the same tokenizer as the encoder:

ix := bm25.NewIndex()
ix.Add("a", "Check-in is at 3 PM.")
ix.Add("b", "Breakfast is served until 10.")
hits := ix.Search("what time is breakfast", 5) // -> [{b, score}]

Sub-100µs search at 100k docs, ~58 MB RAM; build ~46k docs/s. For millions of docs, persistence, or filtering, use Qdrant with the encoder instead.

Performance

Same machine (8-core Apple Silicon), same Qdrant/bm25 model, same corpus (~40 tokens/doc), byte-identical output. Best of 3; full methodology in docs/BENCHMARKS.md.

Throughput — flat from 1k→100k docs (linear scaling); Go is ~3.4×/core and ~12× with parallel batch:

Corpus	Python (1 proc)	Go, 1 core	Go, parallel (8)
1,000	15,196 docs/s	50,783 docs/s	186,563 docs/s
10,000	15,267 docs/s	50,693 docs/s	169,761 docs/s
100,000	15,239 docs/s	51,442 docs/s	188,785 docs/s

Single-encode latency (Go, ~linear in tokens) and cold start (process → first result):

Encode latency	p50	p99		Cold start	time
query (~8 tok)	2.9 µs	6.7 µs		Python FastEmbed	1,435 ms
doc (~40 tok)	14.9 µs	30.4 µs		Go	2.9 ms (~500×)
long (~200 tok)	84.8 µs	172.8 µs

Parallel scaling (100k docs) — no GIL, so near-linear to the 4 performance cores: 1.8× at 2 workers, 3.1× at 4, 3.9× at 8.

EncodeBatchParallel/EncodeStream use all cores; EncodeInto reuses buffers on hot paths. Pure Go (~3 MB binary, no model download) — also runs in WebAssembly.

Languages

New() is English; NewWithLanguage("german") covers the other 16 (SupportedLanguages() lists them, all verified against FastEmbed). WithK, WithB, WithAvgLen, and WithoutStemmer override defaults — though changing them breaks parity with a FastEmbed-indexed corpus.

Compatibility

Identical to FastEmbed Qdrant/bm25 across fastembed 0.4.0–0.8.0 (goldens pinned to 0.7.1; a CI matrix re-checks new releases). Verified with golden vectors captured from FastEmbed and a 1.1M-word stemmer corpus — go test ./....

License

MIT

Documentation

Overview

Package bm25 is a byte-for-byte Go port of Python FastEmbed's "Qdrant/bm25" sparse-text encoder (fastembed==0.7.1).

It turns text into a Qdrant-compatible sparse vector — parallel arrays of token-id indices and BM25 term-frequency values — so a Go service can build the sparse half of a Qdrant hybrid search against a corpus that was indexed in Python with FastEmbed. The token ids match FastEmbed's exactly, so queries line up with the stored vectors.

Usage

Construct an Encoder with New, then call Encoder.Encode:

enc := bm25.New()
indices, values := enc.Encode("What is the cancellation policy?")

indices are uint32 token ids and values are float32 BM25 term-frequency weights; the two slices are aligned and ordered by first occurrence of each unique stemmed token. Use Encoder.EncodeBatch for many texts at once.

Languages

New is English. For other languages use NewWithLanguage, which returns an error for unsupported languages:

enc, err := bm25.NewWithLanguage("german")

SupportedLanguages lists the 16 languages whose output is verified byte-for-byte against FastEmbed (danish, dutch, english, finnish, french, german, hungarian, italian, norwegian, portuguese, romanian, russian, spanish, swedish, tamil, turkish). WithK, WithB, WithAvgLen, and WithoutStemmer override defaults.

Query vs document encoding

Encoder.Encode is the document path (BM25 term-frequency weights), used for indexing and matching a FastEmbed-indexed corpus. Encoder.EncodeQuery is FastEmbed's query_embed path (every token weighted 1.0) for query vectors.

Local search (no Qdrant)

Index is an in-memory BM25 search index — full BM25 ranking with IDF, no external service — using the same tokenizer and stemmer as Encoder. Use it for tests, CLIs, edge/offline use, and small-to-medium corpora; use Qdrant with Encoder for scale, persistence, and filtering.

Concurrency

A single Encoder is safe to share across goroutines. Encoder.EncodeBatchParallel spreads batch encoding over a worker pool (Go has no GIL, so this scales across cores), and Encoder.EncodeStream is a context-cancellable channel pipeline for streaming ETL — both for indexing large corpora.

Performance

On an 8-core machine, encoding is ~3.5x faster per core than Python FastEmbed and ~11.5x with EncodeBatchParallel, with ~500x faster cold start (a single ~3 MB static binary, no model download or ONNX Runtime). The encoder is pure Go and also compiles to WebAssembly (js/wasm and wasip1/wasm). See docs/BENCHMARKS.md and _examples/wasm in the repository.

Term frequencies only (no IDF)

The encoder emits term frequencies only — never IDF. IDF is applied server-side by Qdrant via the sparse vector's IDF modifier, exactly as FastEmbed expects. Do not apply IDF on the client.

Pipeline

Encoding mirrors fastembed.sparse.bm25.Bm25's document path:

lowercase
  -> split on non-word characters (Unicode-aware)
  -> drop stopwords, single-character punctuation, tokens longer than 40 runes
  -> Snowball (English) stemming
  -> BM25 term-frequency weighting (k=1.2, b=0.75, avg_len=256)
  -> abs(murmurhash3_x86_32(token, seed 0)) token ids

Empty, whitespace-only, and stopword-only input return empty slices.

Compatibility

Output is verified byte-for-byte against FastEmbed's "Qdrant/bm25" model across fastembed 0.4.0–0.8.0 (the model output is unchanged across those releases; goldens are pinned to 0.7.1). See the project README for a Qdrant Go client example and the full parity-testing setup: https://github.com/harsh04/bm25

Example

Encode a single query into a Qdrant-compatible sparse vector.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("What is the cancellation policy?")

	// Stopwords ("what", "is", "the") are dropped; "cancellation" and "policy"
	// are stemmed, then hashed to token ids. Values are term frequencies (no IDF).
	fmt.Println("terms:", len(indices))
	for i := range indices {
		fmt.Printf("%d %.4f\n", indices[i], values[i])
	}
}

Output:
terms: 2
1818586924 1.6832
203139330 1.6832

Index

• Constants
• func SupportedLanguages() []string
• type Encoder
  • func New() *Encoder
  • func NewWithLanguage(language string, opts ...Option) (*Encoder, error)
  • func (e *Encoder) Encode(text string) (indices []uint32, values []float32)
  • func (e *Encoder) EncodeBatch(texts []string) (indices [][]uint32, values [][]float32)
  • func (e *Encoder) EncodeBatchParallel(texts []string, workers int) (indices [][]uint32, values [][]float32)
  • func (e *Encoder) EncodeInto(text string, indices []uint32, values []float32) ([]uint32, []float32)
  • func (e *Encoder) EncodeQuery(text string) (indices []uint32, values []float32)
  • func (e *Encoder) EncodeStream(ctx context.Context, texts <-chan string, workers int) <-chan StreamResult
  • func (e *Encoder) Language() string
• type Hit
• type Index
  • func NewIndex() *Index
  • func (ix *Index) Add(id, text string)
  • func (ix *Index) Len() int
  • func (ix *Index) Search(query string, k int) []Hit
• type Option
  • func WithAvgLen(avgLen float64) Option
  • func WithB(b float64) Option
  • func WithK(k float64) Option
  • func WithoutStemmer() Option
• type StreamResult

Examples

• Package
• Encoder.Encode
• Encoder.Encode (StopwordsOnly)
• Encoder.EncodeBatch
• Encoder.EncodeQuery
• Index
• NewWithLanguage

Constants

const (
	DefaultK        = 1.2   // term-frequency saturation
	DefaultB        = 0.75  // document-length normalization
	DefaultAvgLen   = 256.0 // assumed average document length
	DefaultLanguage = "english"
)

BM25 hyperparameters for the "Qdrant/bm25" model. These are fixed (stateless); the encoder does not see a corpus, so avgLen is a constant, not a measured mean.

Functions

func SupportedLanguages

func SupportedLanguages() []string

SupportedLanguages returns the languages this package can encode, sorted. Each one matches FastEmbed's "Qdrant/bm25" output for that language.

Types

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder produces FastEmbed-compatible BM25 sparse vectors. Construct one with New (English) or NewWithLanguage. An Encoder is safe for concurrent use; all fields are read-only after construction, so a single Encoder can be shared across goroutines.

func New

func New() *Encoder

New returns an English encoder with FastEmbed's "Qdrant/bm25" defaults (k=1.2, b=0.75, avgLen=256, English stopwords + Snowball stemmer).

func NewWithLanguage

func NewWithLanguage(language string, opts ...Option) (*Encoder, error)

NewWithLanguage returns an encoder for the given language (see SupportedLanguages), with optional overrides. It returns an error if the language is not supported.

Example

NewWithLanguage builds an encoder for a non-English language.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc, err := bm25.NewWithLanguage("german")
	if err != nil {
		panic(err)
	}
	indices, _ := enc.Encode("Wann ist der Check-in?")
	fmt.Println(len(indices), "terms")
}

Output:
2 terms

func (*Encoder) Encode

func (e *Encoder) Encode(text string) (indices []uint32, values []float32)

Encode turns a single text into a sparse vector: indices are token ids (abs(murmurhash3_x86_32(stemmed_token, seed=0))) and values are the BM25 term-frequency weights. Empty, whitespace-only, and stopword-only text yields empty slices.

The arrays are aligned and ordered by first occurrence of each unique stemmed token, matching FastEmbed's output ordering exactly. Values are term frequencies only; Qdrant applies IDF server-side. To encode many texts, use Encoder.EncodeBatch.

The returned slices are freshly allocated and owned by the caller (safe to retain or mutate), and are non-nil even when empty. For a buffer-reusing variant on hot paths, see Encoder.EncodeInto.

Example

Stemming collapses inflected forms to one term, and repeats are counted together, so "running"/"runs" both fold into a single entry.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, _ := enc.Encode("running runs ran runner")

	// run (from running+runs), ran, runner -> 3 unique terms.
	fmt.Println(len(indices), "unique terms")
}

Output:
3 unique terms

Example (StopwordsOnly)

Empty, whitespace-only, and stopword-only text produce an empty sparse vector.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("the and of is")
	fmt.Println(len(indices), len(values))
}

Output:
0 0

func (*Encoder) EncodeBatch

func (e *Encoder) EncodeBatch(texts []string) (indices [][]uint32, values [][]float32)

EncodeBatch encodes each text independently with Encoder.Encode. The returned slices are aligned with texts: indices[i]/values[i] are the sparse vector for texts[i]. An empty input yields empty (non-nil) results.

Example

EncodeBatch encodes several texts in one call; results are aligned by index.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, _ := enc.EncodeBatch([]string{"wifi password", "free breakfast"})

	for i, idx := range indices {
		fmt.Printf("doc %d: %d terms\n", i, len(idx))
	}
}

Output:
doc 0: 2 terms
doc 1: 2 terms

func (*Encoder) EncodeBatchParallel

func (e *Encoder) EncodeBatchParallel(texts []string, workers int) (indices [][]uint32, values [][]float32)

EncodeBatchParallel is Encoder.EncodeBatch across a worker pool — Go has no GIL, so this scales encoding across all cores in-process, which matters when indexing large corpora. Output is identical to EncodeBatch (results stay aligned with texts by index); only the work is parallelized.

workers <= 0 uses runtime.NumCPU(). A single Encoder is safe to share here.

func (*Encoder) EncodeInto

func (e *Encoder) EncodeInto(text string, indices []uint32, values []float32) ([]uint32, []float32)

EncodeInto is Encoder.Encode that appends into caller-provided buffers, avoiding allocations in hot loops. Pass the slices returned by the previous call (or nil); they are truncated and refilled, and the grown slices returned:

var idx []uint32
var val []float32
for _, q := range queries {
	idx, val = enc.EncodeInto(q, idx, val)
	// use idx/val before the next call — they are overwritten
}

Output is identical to Encoder.Encode; the returned slices are non-nil even when empty.

func (*Encoder) EncodeQuery

func (e *Encoder) EncodeQuery(text string) (indices []uint32, values []float32)

EncodeQuery encodes a query the way FastEmbed's query_embed does: each unique stemmed token gets a weight of 1.0, with no term-frequency weighting. Use this for QUERY vectors when your corpus was indexed with FastEmbed's standard document/query split; use Encoder.Encode (term frequencies) when both your index and queries go through the document path.

Indices are deduplicated token ids in first-occurrence order and values are all 1.0. (FastEmbed emits them in unspecified set order; Qdrant treats a sparse vector as a map, so order does not affect retrieval.) The returned slices are non-nil even when empty.

Example

EncodeQuery weights every unique token 1.0 (FastEmbed's query_embed path), rather than by term frequency.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	_, values := enc.EncodeQuery("breakfast breakfast pool")
	fmt.Println(values)
}

Output:
[1 1]

func (*Encoder) EncodeStream

func (e *Encoder) EncodeStream(ctx context.Context, texts <-chan string, workers int) <-chan StreamResult

EncodeStream encodes a stream of texts concurrently, emitting results as they finish (order not preserved — use Text to correlate). It's a building block for pipelined ETL: feed documents in, upsert sparse vectors out, with backpressure and cancellation via ctx.

The output channel is closed when texts is drained or ctx is cancelled. workers <= 0 uses runtime.NumCPU().

func (*Encoder) Language

func (e *Encoder) Language() string

Language returns the encoder's configured language.

type Hit

type Hit struct {
	ID    string
	Score float64
}

Hit is a single search result, ranked by descending BM25 score.

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index is an in-memory BM25 search index — full BM25 ranking (with IDF) over a corpus you hold in memory, with no external service. It uses the same tokenizer and stemmer as Encoder, so a document indexed here produces the same tokens it would in Qdrant; you can prototype retrieval locally and move the identical tokenization to a Qdrant sparse vector later.

Use it for tests, CLIs, edge/offline use, notebooks, and small-to-medium corpora where standing up Qdrant is overkill. For millions of documents, persistence, filtering, or distribution, use Qdrant with Encoder.

An Index is not safe for concurrent Add; build it, then it is safe for concurrent Search.

Example

Index does full BM25 ranking (with IDF) in memory, no Qdrant needed.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	ix := bm25.NewIndex()
	ix.Add("a", "Check-in is at 3 PM.")
	ix.Add("b", "Breakfast is served until 10.")
	ix.Add("c", "The pool is open until 9 PM.")

	for _, hit := range ix.Search("what time is breakfast", 1) {
		fmt.Println(hit.ID)
	}
}

Output:
b

func NewIndex

func NewIndex() *Index

NewIndex returns an empty English BM25 index with FastEmbed's default parameters. (Other languages can be added later via the encoder config.)

func (*Index) Add

func (ix *Index) Add(id, text string)

Add indexes a document under the given id (returned in search results). The text is tokenized and stemmed exactly as Encoder.Encode would.

func (*Index) Len

func (ix *Index) Len() int

Len reports the number of indexed documents.

func (*Index) Search

func (ix *Index) Search(query string, k int) []Hit

Search returns up to k documents ranked by BM25 score (Okapi BM25 with Lucene/Qdrant-style IDF). It returns nil if k <= 0, the index is empty, or the query has no indexable terms.

type Option

type Option func(*config)

Option configures an Encoder built by NewWithLanguage. The set of options is closed by design; use WithK, WithB, WithAvgLen, or WithoutStemmer.

func WithAvgLen

func WithAvgLen(avgLen float64) Option

WithAvgLen overrides the assumed average document length (default DefaultAvgLen). See WithK for the parity caveat.

func WithB

func WithB(b float64) Option

WithB overrides the BM25 document-length normalization parameter (default DefaultB). See WithK for the parity caveat.

func WithK

func WithK(k float64) Option

WithK overrides the BM25 term-frequency saturation parameter (default DefaultK). Changing k/b/avgLen breaks parity with a FastEmbed-indexed corpus — leave them alone unless you are building an independent index.

func WithoutStemmer

func WithoutStemmer() Option

WithoutStemmer disables stemming. This mirrors FastEmbed's disable_stemmer=True, which also drops stopword removal — tokens are only lowercased, filtered by length/punctuation, then hashed.

type StreamResult

type StreamResult struct {
	Text    string
	Indices []uint32
	Values  []float32
}

StreamResult is one encoded document from Encoder.EncodeStream. Text echoes the input so callers can correlate results, which may arrive out of order.