bm25

README

bm25 — FastEmbed Qdrant/bm25 sparse encoder for Go

BM25 sparse vectors in Go for Qdrant hybrid search, matching Python FastEmbed byte-for-byte.

A Go port of FastEmbed's Qdrant/bm25 sparse text encoder (aka fastembed-bm25-go). It builds the BM25 sparse vectors a Go service needs for Qdrant hybrid search, and its output matches the Python encoder exactly — so the token ids line up with a corpus that was indexed in Python with FastEmbed. It's the sparse counterpart to the dense fastembed-go port.

Why a Go BM25 encoder for Qdrant

Qdrant hybrid search combines a dense vector with a sparse BM25 vector and fuses the two result sets. The sparse vectors are usually produced in Python by FastEmbed's SparseTextEmbedding("Qdrant/bm25"). Each token id in a sparse vector is a hash of a stemmed token, so a query only matches the stored corpus if it is tokenized, stemmed, and hashed in exactly the same way.

There was no Go encoder that did this. The dense FastEmbed port, anush008/fastembed-go, does not cover sparse models. This package fills that gap and is checked against output captured from FastEmbed itself.

Why this library vs. Qdrant server-side BM25

Since Qdrant v1.15.2, Qdrant can convert text to Qdrant/bm25 sparse vectors server-side (via the Inference API / Qdrant Cloud Inference), and the official Go client can pass Document{Model: "qdrant/bm25", Text: ...}. So why compute BM25 vectors in Go? Use this library when you want:

• Client-side control — tokenization and vectorization happen entirely in your Go process: no Inference API, no extra network round-trip, no dependency on a Qdrant version or feature flag.
• Self-hosted or older Qdrant — works against any Qdrant, or none.
• Byte-for-byte FastEmbed parity — produces the same sparse vectors as Python FastEmbed's Qdrant/bm25, so a Go service can query a corpus indexed by a Python/FastEmbed pipeline. Verified with golden tests.
• Non-Qdrant / offline use — plain BM25 sparse vectors for any sparse retrieval backend, or for testing and evaluation without a server.

Install

go get github.com/harsh04/bm25

Building BM25 sparse query vectors in Go

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("What is the cancellation policy?")
	fmt.Println(indices, values)
}

indices are uint32 token ids and values are float32 BM25 term-frequency weights. The two slices are aligned and ordered by first occurrence of each unique stemmed token.

Languages

New() is English. For other languages use NewWithLanguage:

enc, err := bm25.NewWithLanguage("german")
indices, values := enc.Encode("Wann ist die Stornierungsfrist?")

SupportedLanguages() returns the 15 languages whose output is verified byte-for-byte against FastEmbed: danish, dutch, english, finnish, french, german, hungarian, italian, norwegian, portuguese, romanian, russian, spanish, swedish, turkish. WithParams(k, b, avgLen) and WithoutStemmer() override the defaults.

Qdrant hybrid search (dense + sparse vectors)

Use the sparse vector from this package as the sparse half of a hybrid query with the Qdrant Go client:

indices, values := enc.Encode(query)

sparse := &qdrant.Vector{
	Indices: &qdrant.SparseIndices{Data: indices},
	Data:    values,
}

(Field names vary slightly between client versions; adjust as needed.) Pair this with a dense embedding and Qdrant's RRF fusion to assemble full hybrid search.

Scope

• Term frequencies only. IDF is applied by Qdrant on the server through the sparse vector's IDF modifier, the way FastEmbed expects. Do not apply IDF on the client.
• Stateless. k = 1.2, b = 0.75, and an assumed average length of 256 are fixed, matching the model. No corpus statistics are needed.
• Empty, whitespace-only, and stopword-only input return empty slices.

FastEmbed Go compatibility

Tracks fastembed==0.7.1 (model Qdrant/bm25). The pipeline is:

lowercase
  → split on non-word characters
  → drop stopwords, single-character punctuation, and tokens longer than 40 chars
  → Snowball (English) stemming
  → BM25 term-frequency weighting
  → abs(murmurhash3_x86_32(token, seed 0)) token ids

How parity is tested

Parity is verified against data captured from FastEmbed rather than assumed:

• testdata/golden_vectors.json — 50 queries (punctuation, repeated tokens, mixed scripts, emoji, long tokens, stopword-only, underscores) with the exact indices and values FastEmbed produces.
• testdata/intermediate_stages.json — per-stage output (tokens, stems, term-frequency map) for debugging individual cases.
• testdata/stemmer_golden.json — 12,469 English word/stem pairs from py_rust_stemmers; the Go stemmer (blevesearch/snowballstem) matches all of them, and all 15 languages match across 700k+ words of the Snowball test vocab.
• testdata/multilang_goldens.json — per-language encode goldens for all 15 supported languages.

go test ./...

Regenerate the fixtures (requires Python with fastembed==0.7.1):

python tools/gen_goldens.py            # English vectors + stemmer corpus
python tools/gen_multilang_goldens.py  # per-language vectors

Notes

• Python's regex \w is Unicode-aware and treats combining marks as separators (for example, Devanagari and Tamil vowel signs split a word). Go's regexp \w is ASCII only, so tokenization is done by hand to match.
• mmh3.hash returns a signed 32-bit value; token ids are its absolute value. abs(-2^31) is 2^31, which does not fit in int32 but does fit in uint32.

Dependencies

• github.com/blevesearch/snowballstem — Snowball stemmers
• github.com/twmb/murmur3 — MurmurHash3

License

MIT

Documentation

Overview

Package bm25 is a byte-for-byte Go port of Python FastEmbed's "Qdrant/bm25" sparse-text encoder (fastembed==0.7.1).

It turns text into a Qdrant-compatible sparse vector — parallel arrays of token-id indices and BM25 term-frequency values — so a Go service can build the sparse half of a Qdrant hybrid search against a corpus that was indexed in Python with FastEmbed. The token ids match FastEmbed's exactly, so queries line up with the stored vectors.

Usage

Construct an Encoder with New, then call Encoder.Encode:

enc := bm25.New()
indices, values := enc.Encode("What is the cancellation policy?")

indices are uint32 token ids and values are float32 BM25 term-frequency weights; the two slices are aligned and ordered by first occurrence of each unique stemmed token. Use Encoder.EncodeBatch for many texts at once.

Languages

New is English. For other languages use NewWithLanguage, which returns an error for unsupported languages:

enc, err := bm25.NewWithLanguage("german")

SupportedLanguages lists the 15 languages whose output is verified byte-for-byte against FastEmbed (danish, dutch, english, finnish, french, german, hungarian, italian, norwegian, portuguese, romanian, russian, spanish, swedish, turkish). WithParams and WithoutStemmer override defaults.

Term frequencies only (no IDF)

The encoder emits term frequencies only — never IDF. IDF is applied server-side by Qdrant via the sparse vector's IDF modifier, exactly as FastEmbed expects. Do not apply IDF on the client.

Pipeline

Encoding mirrors fastembed.sparse.bm25.Bm25's document path:

lowercase
  -> split on non-word characters (Unicode-aware)
  -> drop stopwords, single-character punctuation, tokens longer than 40 runes
  -> Snowball (English) stemming
  -> BM25 term-frequency weighting (k=1.2, b=0.75, avg_len=256)
  -> abs(murmurhash3_x86_32(token, seed 0)) token ids

Empty, whitespace-only, and stopword-only input return empty slices.

Compatibility

Output is verified byte-for-byte against fastembed==0.7.1 (model "Qdrant/bm25") with golden tests. See the project README for a Qdrant Go client example and the full parity-testing setup: https://github.com/harsh04/bm25

Example

Encode a single query into a Qdrant-compatible sparse vector.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("What is the cancellation policy?")

	// Stopwords ("what", "is", "the") are dropped; "cancellation" and "policy"
	// are stemmed, then hashed to token ids. Values are term frequencies (no IDF).
	fmt.Println("terms:", len(indices))
	for i := range indices {
		fmt.Printf("%d %.4f\n", indices[i], values[i])
	}
}

Output:
terms: 2
1818586924 1.6832
203139330 1.6832

Index

• Constants
• func SupportedLanguages() []string
• type Encoder
  • func New() *Encoder
  • func NewWithLanguage(language string, opts ...Option) (*Encoder, error)
  • func (e *Encoder) Encode(text string) (indices []uint32, values []float32)
  • func (e *Encoder) EncodeBatch(texts []string) (indices [][]uint32, values [][]float32)
• type Option
  • func WithParams(k, b, avgLen float64) Option
  • func WithoutStemmer() Option

Examples

• Package
• Encoder.Encode
• Encoder.Encode (StopwordsOnly)
• Encoder.EncodeBatch
• NewWithLanguage

Constants

const (
	DefaultK        = 1.2   // term-frequency saturation
	DefaultB        = 0.75  // document-length normalization
	DefaultAvgLen   = 256.0 // assumed average document length
	DefaultLanguage = "english"
)

BM25 hyperparameters for the "Qdrant/bm25" model. These are fixed (stateless); the encoder does not see a corpus, so avgLen is a constant, not a measured mean.

Functions

func SupportedLanguages

func SupportedLanguages() []string

SupportedLanguages returns the languages this package can encode, sorted. Each one matches FastEmbed's "Qdrant/bm25" output for that language.

Types

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder produces FastEmbed-compatible BM25 sparse vectors. Construct one with New (English) or NewWithLanguage. An Encoder is safe for concurrent use; all fields are read-only after construction, so a single Encoder can be shared across goroutines.

func New

func New() *Encoder

New returns an English encoder with FastEmbed's "Qdrant/bm25" defaults (k=1.2, b=0.75, avgLen=256, English stopwords + Snowball stemmer).

func NewWithLanguage

func NewWithLanguage(language string, opts ...Option) (*Encoder, error)

NewWithLanguage returns an encoder for the given language (see SupportedLanguages), with optional overrides. It returns an error if the language is not supported.

Example

NewWithLanguage builds an encoder for a non-English language.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc, err := bm25.NewWithLanguage("german")
	if err != nil {
		panic(err)
	}
	indices, _ := enc.Encode("Wann ist der Check-in?")
	fmt.Println(len(indices), "terms")
}

Output:
2 terms

func (*Encoder) Encode

func (e *Encoder) Encode(text string) (indices []uint32, values []float32)

Encode turns a single text into a sparse vector: indices are token ids (abs(murmurhash3_x86_32(stemmed_token, seed=0))) and values are the BM25 term-frequency weights. Empty, whitespace-only, and stopword-only text yields empty slices.

The arrays are aligned and ordered by first occurrence of each unique stemmed token, matching FastEmbed's output ordering exactly. Values are term frequencies only; Qdrant applies IDF server-side. To encode many texts, use Encoder.EncodeBatch.

Example

Stemming collapses inflected forms to one term, and repeats are counted together, so "running"/"runs" both fold into a single entry.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, _ := enc.Encode("running runs ran runner")

	// run (from running+runs), ran, runner -> 3 unique terms.
	fmt.Println(len(indices), "unique terms")
}

Output:
3 unique terms

Example (StopwordsOnly)

Empty, whitespace-only, and stopword-only text produce an empty sparse vector.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("the and of is")
	fmt.Println(len(indices), len(values))
}

Output:
0 0

func (*Encoder) EncodeBatch

func (e *Encoder) EncodeBatch(texts []string) (indices [][]uint32, values [][]float32)

EncodeBatch encodes each text independently with Encoder.Encode. The returned slices are aligned with texts: indices[i]/values[i] are the sparse vector for texts[i]. An empty input yields empty (non-nil) results.

Example

EncodeBatch encodes several texts in one call; results are aligned by index.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, _ := enc.EncodeBatch([]string{"wifi password", "free breakfast"})

	for i, idx := range indices {
		fmt.Printf("doc %d: %d terms\n", i, len(idx))
	}
}

Output:
doc 0: 2 terms
doc 1: 2 terms

type Option

type Option func(*config)

Option configures an Encoder built by NewWithLanguage.

func WithParams

func WithParams(k, b, avgLen float64) Option

WithParams overrides the BM25 hyperparameters. The defaults (DefaultK, DefaultB, DefaultAvgLen) match FastEmbed's "Qdrant/bm25" and should be left alone for parity with a FastEmbed-indexed corpus.

func WithoutStemmer

func WithoutStemmer() Option

WithoutStemmer disables stemming. This mirrors FastEmbed's disable_stemmer=True, which also drops stopword removal — tokens are only lowercased, filtered by length/punctuation, then hashed.