bench

Documentation

Overview

Package bench is turbograph's retrieval benchmark harness: dataset loaders, a deterministic offline embedder for regression gating, and an evaluation runner that scores the real ingestion and retrieval pipeline. The same code powers the committed CI regression suite (offline, deterministic) and the `turbograph bench` command that reproduces the headline numbers with a real model.

Index

• func Evaluate(ctx context.Context, embedder rag.Embedder, cfg rag.Config, ds *Dataset, ...) (eval.Report, error)
• func LoadSuiteFile(path string) ([]eval.Case, error)
• type Dataset
  • func LoadBEIR(corpusPath, queriesPath, qrelsPath string) (*Dataset, error)
• type HashEmbedder
  • func (e HashEmbedder) Embed(_ context.Context, texts []string) ([][]float32, error)
• type Options

Functions

func Evaluate

func Evaluate(ctx context.Context, embedder rag.Embedder, cfg rag.Config, ds *Dataset, opt Options) (eval.Report, error)

Evaluate ingests a dataset into a fresh store built on embedder and cfg, then scores retrieval for every labeled query. With DocLevel set, the ranking collapses chunks to their document ids before scoring, which matches BEIR document-level qrels; otherwise chunk ids are scored directly.

func LoadSuiteFile

func LoadSuiteFile(path string) ([]eval.Case, error)

LoadSuiteFile loads a turbograph eval suite (JSONL of {query, relevant}) and the corpus separately, for chunk-level or custom datasets.

Types

type Dataset

type Dataset struct {
	Name  string
	Docs  []rag.Document
	Cases []eval.Case
}

Dataset is a loaded benchmark: the corpus to ingest and the labeled queries to score against, with relevance keyed by document id.

func LoadBEIR

func LoadBEIR(corpusPath, queriesPath, qrelsPath string) (*Dataset, error)

LoadBEIR loads a dataset in the BEIR layout: corpus.jsonl ({"_id","title", "text"}), queries.jsonl ({"_id","text"}), and a qrels TSV (query-id<TAB>corpus-id<TAB>score, score >= 1 meaning relevant, with an optional header row). Relevance is at the document level, the BEIR convention.

type HashEmbedder

type HashEmbedder struct {
	Dim int // embedding dimension; 256 is a good default for the tests
}

HashEmbedder is a deterministic, dependency-free bag-of-words embedder used for offline regression tests. Each token is hashed into the vector with a signed hashing trick and weighted by term frequency, then the vector is L2-normalized. It is not a semantic model: cosine similarity reflects shared vocabulary, which is enough to verify that the retrieval pipeline ranks lexically-related text correctly and to catch regressions without a network or a model server. It is never used for the published benchmark numbers, which use a real embedder.

func (HashEmbedder) Embed

func (e HashEmbedder) Embed(_ context.Context, texts []string) ([][]float32, error)

Embed implements rag.Embedder.

type Options

type Options struct {
	K          int                // cutoff for the @k metrics (default 10)
	DocLevel   bool               // collapse retrieved chunks to documents (BEIR convention)
	Params     rag.RetrieveParams // retrieval knobs; TopK defaults to max(K, 10)
	OnProgress func(done, total int)
}

Options configures an evaluation run.