bm25

README

bm25

A Go port of FastEmbed's Qdrant/bm25 sparse text encoder. Its output matches the Python encoder exactly, so a Go service can build the sparse vectors for Qdrant hybrid search and query a corpus that was indexed with FastEmbed.

There wasn't a Go equivalent — the dense fastembed-go port doesn't cover sparse models. Sparse token ids are hashes of stemmed tokens, so a query only lines up with the stored corpus if it's tokenized, stemmed, and hashed the same way. This does, tested against output captured from FastEmbed itself.

Qdrant can also produce these server-side (since v1.15.2) — reach for this when you want encoding client-side, on older or self-hosted Qdrant, or matching a corpus already indexed by Python FastEmbed.

Install

go get github.com/harsh04/bm25

Usage

enc := bm25.New()
indices, values := enc.Encode("What is the cancellation policy?")
// indices []uint32 (token ids), values []float32 (term-frequency weights)

The encoder emits term frequencies only — Qdrant applies IDF server-side via the sparse vector's modifier, so don't apply it yourself. Empty or stopword-only text returns empty slices, and an Encoder is safe to share across goroutines.

To run a query, pass the indices/values to the Qdrant Go client's NewVectorSparse / NewQuerySparse. There's a full dense+sparse+RRF example in _examples/qdrant_hybrid, and the qhybrid module wraps the whole hybrid flow if you'd rather not wire it yourself.

Languages

New() is English; NewWithLanguage("german") covers the other 15 (SupportedLanguages() lists them, all verified against FastEmbed). WithK, WithB, WithAvgLen, and WithoutStemmer override the defaults — though changing them breaks parity with a FastEmbed-indexed corpus.

Performance

Same machine and corpus, byte-identical output (methodology):

	Python FastEmbed	Go
Throughput, 1 core	14.8k docs/sec	51k (~3.5×)
Throughput, 8 cores	—	170k (~11.5×)
Cold start → first result	1,435 ms	2.9 ms (~500×)

EncodeBatchParallel and EncodeStream spread work across cores (no GIL); EncodeInto reuses buffers on hot paths. It's a ~3 MB static binary with no model download, and compiles to WebAssembly for the browser or edge (_examples/wasm).

Compatibility

Output is identical to FastEmbed Qdrant/bm25 across fastembed 0.4.0–0.8.0 (goldens pinned to 0.7.1; a CI matrix re-checks new releases). Parity is verified with golden vectors captured from FastEmbed and a 1.1M-word stemmer corpus — run go test ./....

License

MIT

Documentation

Overview

Package bm25 is a byte-for-byte Go port of Python FastEmbed's "Qdrant/bm25" sparse-text encoder (fastembed==0.7.1).

It turns text into a Qdrant-compatible sparse vector — parallel arrays of token-id indices and BM25 term-frequency values — so a Go service can build the sparse half of a Qdrant hybrid search against a corpus that was indexed in Python with FastEmbed. The token ids match FastEmbed's exactly, so queries line up with the stored vectors.

Usage

Construct an Encoder with New, then call Encoder.Encode:

enc := bm25.New()
indices, values := enc.Encode("What is the cancellation policy?")

indices are uint32 token ids and values are float32 BM25 term-frequency weights; the two slices are aligned and ordered by first occurrence of each unique stemmed token. Use Encoder.EncodeBatch for many texts at once.

Languages

New is English. For other languages use NewWithLanguage, which returns an error for unsupported languages:

enc, err := bm25.NewWithLanguage("german")

SupportedLanguages lists the 16 languages whose output is verified byte-for-byte against FastEmbed (danish, dutch, english, finnish, french, german, hungarian, italian, norwegian, portuguese, romanian, russian, spanish, swedish, tamil, turkish). WithK, WithB, WithAvgLen, and WithoutStemmer override defaults.

Query vs document encoding

Encoder.Encode is the document path (BM25 term-frequency weights), used for indexing and matching a FastEmbed-indexed corpus. Encoder.EncodeQuery is FastEmbed's query_embed path (every token weighted 1.0) for query vectors.

Concurrency

A single Encoder is safe to share across goroutines. Encoder.EncodeBatchParallel spreads batch encoding over a worker pool (Go has no GIL, so this scales across cores), and Encoder.EncodeStream is a context-cancellable channel pipeline for streaming ETL — both for indexing large corpora.

Performance

On an 8-core machine, encoding is ~3.5x faster per core than Python FastEmbed and ~11.5x with EncodeBatchParallel, with ~500x faster cold start (a single ~3 MB static binary, no model download or ONNX Runtime). The encoder is pure Go and also compiles to WebAssembly (js/wasm and wasip1/wasm). See docs/BENCHMARKS.md and _examples/wasm in the repository.

Term frequencies only (no IDF)

The encoder emits term frequencies only — never IDF. IDF is applied server-side by Qdrant via the sparse vector's IDF modifier, exactly as FastEmbed expects. Do not apply IDF on the client.

Pipeline

Encoding mirrors fastembed.sparse.bm25.Bm25's document path:

lowercase
  -> split on non-word characters (Unicode-aware)
  -> drop stopwords, single-character punctuation, tokens longer than 40 runes
  -> Snowball (English) stemming
  -> BM25 term-frequency weighting (k=1.2, b=0.75, avg_len=256)
  -> abs(murmurhash3_x86_32(token, seed 0)) token ids

Empty, whitespace-only, and stopword-only input return empty slices.

Compatibility

Output is verified byte-for-byte against FastEmbed's "Qdrant/bm25" model across fastembed 0.4.0–0.8.0 (the model output is unchanged across those releases; goldens are pinned to 0.7.1). See the project README for a Qdrant Go client example and the full parity-testing setup: https://github.com/harsh04/bm25

Example

Encode a single query into a Qdrant-compatible sparse vector.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("What is the cancellation policy?")

	// Stopwords ("what", "is", "the") are dropped; "cancellation" and "policy"
	// are stemmed, then hashed to token ids. Values are term frequencies (no IDF).
	fmt.Println("terms:", len(indices))
	for i := range indices {
		fmt.Printf("%d %.4f\n", indices[i], values[i])
	}
}

Output:
terms: 2
1818586924 1.6832
203139330 1.6832

Index

• Constants
• func SupportedLanguages() []string
• type Encoder
  • func New() *Encoder
  • func NewWithLanguage(language string, opts ...Option) (*Encoder, error)
  • func (e *Encoder) Encode(text string) (indices []uint32, values []float32)
  • func (e *Encoder) EncodeBatch(texts []string) (indices [][]uint32, values [][]float32)
  • func (e *Encoder) EncodeBatchParallel(texts []string, workers int) (indices [][]uint32, values [][]float32)
  • func (e *Encoder) EncodeInto(text string, indices []uint32, values []float32) ([]uint32, []float32)
  • func (e *Encoder) EncodeQuery(text string) (indices []uint32, values []float32)
  • func (e *Encoder) EncodeStream(ctx context.Context, texts <-chan string, workers int) <-chan StreamResult
  • func (e *Encoder) Language() string
• type Option
  • func WithAvgLen(avgLen float64) Option
  • func WithB(b float64) Option
  • func WithK(k float64) Option
  • func WithoutStemmer() Option
• type StreamResult

Examples

• Package
• Encoder.Encode
• Encoder.Encode (StopwordsOnly)
• Encoder.EncodeBatch
• Encoder.EncodeQuery
• NewWithLanguage

Constants

const (
	DefaultK        = 1.2   // term-frequency saturation
	DefaultB        = 0.75  // document-length normalization
	DefaultAvgLen   = 256.0 // assumed average document length
	DefaultLanguage = "english"
)

BM25 hyperparameters for the "Qdrant/bm25" model. These are fixed (stateless); the encoder does not see a corpus, so avgLen is a constant, not a measured mean.

Functions

func SupportedLanguages

func SupportedLanguages() []string

SupportedLanguages returns the languages this package can encode, sorted. Each one matches FastEmbed's "Qdrant/bm25" output for that language.

Types

type Encoder

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder produces FastEmbed-compatible BM25 sparse vectors. Construct one with New (English) or NewWithLanguage. An Encoder is safe for concurrent use; all fields are read-only after construction, so a single Encoder can be shared across goroutines.

func New

func New() *Encoder

New returns an English encoder with FastEmbed's "Qdrant/bm25" defaults (k=1.2, b=0.75, avgLen=256, English stopwords + Snowball stemmer).

func NewWithLanguage

func NewWithLanguage(language string, opts ...Option) (*Encoder, error)

NewWithLanguage returns an encoder for the given language (see SupportedLanguages), with optional overrides. It returns an error if the language is not supported.

Example

NewWithLanguage builds an encoder for a non-English language.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc, err := bm25.NewWithLanguage("german")
	if err != nil {
		panic(err)
	}
	indices, _ := enc.Encode("Wann ist der Check-in?")
	fmt.Println(len(indices), "terms")
}

Output:
2 terms

func (*Encoder) Encode

func (e *Encoder) Encode(text string) (indices []uint32, values []float32)

Encode turns a single text into a sparse vector: indices are token ids (abs(murmurhash3_x86_32(stemmed_token, seed=0))) and values are the BM25 term-frequency weights. Empty, whitespace-only, and stopword-only text yields empty slices.

The arrays are aligned and ordered by first occurrence of each unique stemmed token, matching FastEmbed's output ordering exactly. Values are term frequencies only; Qdrant applies IDF server-side. To encode many texts, use Encoder.EncodeBatch.

The returned slices are freshly allocated and owned by the caller (safe to retain or mutate), and are non-nil even when empty. For a buffer-reusing variant on hot paths, see Encoder.EncodeInto.

Example

Stemming collapses inflected forms to one term, and repeats are counted together, so "running"/"runs" both fold into a single entry.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, _ := enc.Encode("running runs ran runner")

	// run (from running+runs), ran, runner -> 3 unique terms.
	fmt.Println(len(indices), "unique terms")
}

Output:
3 unique terms

Example (StopwordsOnly)

Empty, whitespace-only, and stopword-only text produce an empty sparse vector.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, values := enc.Encode("the and of is")
	fmt.Println(len(indices), len(values))
}

Output:
0 0

func (*Encoder) EncodeBatch

func (e *Encoder) EncodeBatch(texts []string) (indices [][]uint32, values [][]float32)

EncodeBatch encodes each text independently with Encoder.Encode. The returned slices are aligned with texts: indices[i]/values[i] are the sparse vector for texts[i]. An empty input yields empty (non-nil) results.

Example

EncodeBatch encodes several texts in one call; results are aligned by index.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	indices, _ := enc.EncodeBatch([]string{"wifi password", "free breakfast"})

	for i, idx := range indices {
		fmt.Printf("doc %d: %d terms\n", i, len(idx))
	}
}

Output:
doc 0: 2 terms
doc 1: 2 terms

func (*Encoder) EncodeBatchParallel

func (e *Encoder) EncodeBatchParallel(texts []string, workers int) (indices [][]uint32, values [][]float32)

EncodeBatchParallel is Encoder.EncodeBatch across a worker pool — Go has no GIL, so this scales encoding across all cores in-process, which matters when indexing large corpora. Output is identical to EncodeBatch (results stay aligned with texts by index); only the work is parallelized.

workers <= 0 uses runtime.NumCPU(). A single Encoder is safe to share here.

func (*Encoder) EncodeInto

func (e *Encoder) EncodeInto(text string, indices []uint32, values []float32) ([]uint32, []float32)

EncodeInto is Encoder.Encode that appends into caller-provided buffers, avoiding allocations in hot loops. Pass the slices returned by the previous call (or nil); they are truncated and refilled, and the grown slices returned:

var idx []uint32
var val []float32
for _, q := range queries {
	idx, val = enc.EncodeInto(q, idx, val)
	// use idx/val before the next call — they are overwritten
}

Output is identical to Encoder.Encode; the returned slices are non-nil even when empty.

func (*Encoder) EncodeQuery

func (e *Encoder) EncodeQuery(text string) (indices []uint32, values []float32)

EncodeQuery encodes a query the way FastEmbed's query_embed does: each unique stemmed token gets a weight of 1.0, with no term-frequency weighting. Use this for QUERY vectors when your corpus was indexed with FastEmbed's standard document/query split; use Encoder.Encode (term frequencies) when both your index and queries go through the document path.

Indices are deduplicated token ids in first-occurrence order and values are all 1.0. (FastEmbed emits them in unspecified set order; Qdrant treats a sparse vector as a map, so order does not affect retrieval.) The returned slices are non-nil even when empty.

Example

EncodeQuery weights every unique token 1.0 (FastEmbed's query_embed path), rather than by term frequency.

package main

import (
	"fmt"

	"github.com/harsh04/bm25"
)

func main() {
	enc := bm25.New()
	_, values := enc.EncodeQuery("breakfast breakfast pool")
	fmt.Println(values)
}

Output:
[1 1]

func (*Encoder) EncodeStream

func (e *Encoder) EncodeStream(ctx context.Context, texts <-chan string, workers int) <-chan StreamResult

EncodeStream encodes a stream of texts concurrently, emitting results as they finish (order not preserved — use Text to correlate). It's a building block for pipelined ETL: feed documents in, upsert sparse vectors out, with backpressure and cancellation via ctx.

The output channel is closed when texts is drained or ctx is cancelled. workers <= 0 uses runtime.NumCPU().

func (*Encoder) Language

func (e *Encoder) Language() string

Language returns the encoder's configured language.

type Option

type Option func(*config)

Option configures an Encoder built by NewWithLanguage. The set of options is closed by design; use WithK, WithB, WithAvgLen, or WithoutStemmer.

func WithAvgLen

func WithAvgLen(avgLen float64) Option

WithAvgLen overrides the assumed average document length (default DefaultAvgLen). See WithK for the parity caveat.

func WithB

func WithB(b float64) Option

WithB overrides the BM25 document-length normalization parameter (default DefaultB). See WithK for the parity caveat.

func WithK

func WithK(k float64) Option

WithK overrides the BM25 term-frequency saturation parameter (default DefaultK). Changing k/b/avgLen breaks parity with a FastEmbed-indexed corpus — leave them alone unless you are building an independent index.

func WithoutStemmer

func WithoutStemmer() Option

WithoutStemmer disables stemming. This mirrors FastEmbed's disable_stemmer=True, which also drops stopword removal — tokens are only lowercased, filtered by length/punctuation, then hashed.

type StreamResult

type StreamResult struct {
	Text    string
	Indices []uint32
	Values  []float32
}

StreamResult is one encoded document from Encoder.EncodeStream. Text echoes the input so callers can correlate results, which may arrive out of order.