bench

Documentation

Overview

Package bench is a reproducible recall + latency harness for aikit's dense retrieval indexes — Flat (exact), HNSW (approximate), and FlatI8 (int8). It turns "parity-tested" into concrete, comparable numbers: recall@k measured against the exact Flat scan, per-query latency percentiles (p50/p95/p99), build time, and index memory. Run it over your own corpus, or use the harness tests (bench_test.go) for the synthetic-scale and real-embedding tables.

It is a benchmarking tool, not a retrieval primitive — Experimental, outside the 1.0 compatibility guarantee.

Index

• func AppendRecords(path string, recs ...Record) error
• func Percentiles(latenciesMs []float64) (p50, p95, p99, mean float64)
• func RecallAt(got []ann.Hit, truth map[int]bool) float64
• func Report(recs []Record) string
• func Table(results []Result) string
• func TruthSet(hits []ann.Hit) map[int]bool
• type Device
  • func HostDevice() Device
• type Meta
  • func CaptureMeta() Meta
• type Quality
• type Record
  • func LoadRecords(path string) ([]Record, error)
• type Result
  • func Run(corpus, queries [][]float32, k int, cfg ann.Config) []Result
• type Shape
• type Timing

Functions

func AppendRecords

func AppendRecords(path string, recs ...Record) error

AppendRecords appends recs to a JSONL file (one compact JSON object per line), creating it if absent. Append-only so a periodic run adds to the corpus rather than clobbering prior machines.

func Percentiles

func Percentiles(latenciesMs []float64) (p50, p95, p99, mean float64)

Percentiles returns p50/p95/p99/mean (ms) of the latencies; it sorts a copy, so the caller's slice is untouched.

func RecallAt

func RecallAt(got []ann.Hit, truth map[int]bool) float64

RecallAt returns |got ∩ truth| / |truth| — mean recall@k of a hit list against the exact-CPU top-k index set (idxSet of ann.New(...).Query). Truth is the ground-truth index set.

func Report

func Report(recs []Record) string

Report renders the full results document from records. Deterministic: everything is sorted, and the only volatile datum embedded is the aikit commit (not a wall-clock time), so re-generating unchanged records yields a byte-identical doc.

func Table

func Table(results []Result) string

Table renders results as a GitHub-flavored Markdown table — paste-ready for a README or a benchmark report.

func TruthSet

func TruthSet(hits []ann.Hit) map[int]bool

TruthSet is the exact-CPU top-k index set for one query, the recall ground truth.

Types

type Device

type Device struct {
	Machine    string `json:"machine"` // stable per-box label, e.g. "apple-m1pro-14c" / "nvidia-2070s"
	Chip       string `json:"chip"`    // CPU chip (baseline) or host chip
	GPU        string `json:"gpu"`     // "" for cpu-simd
	Driver     string `json:"driver,omitempty"`
	SMorFamily string `json:"sm_or_family,omitempty"`
	VRAMMB     int    `json:"vram_mb,omitempty"`
	GOARCH     string `json:"goarch"`
	Cores      int    `json:"cores"`
}

Device is the run's hardware. Machine is the per-box grouping key the report tables split on (two different chips/machines must never share an absolute-numbers column); the rest is the device spec docs/BENCH-gpu.md requires so no cross-machine extrapolation is silent.

func HostDevice

func HostDevice() Device

HostDevice fills the host-derivable device fields (GOARCH, logical cores). The caller sets Machine/Chip and, for a GPU record, GPU/Driver/SMorFamily/VRAMMB — the harness knows the box.

type Meta

type Meta struct {
	AikitCommit    string `json:"aikit_commit"`
	Go             string `json:"go"`
	BuildFlags     string `json:"build_flags,omitempty"`
	Seed           int64  `json:"seed,omitempty"`
	WarmupLaunches int    `json:"warmup_launches,omitempty"`
	Iters          int    `json:"iters,omitempty"`
}

Meta is the reproducibility context.

func CaptureMeta

func CaptureMeta() Meta

CaptureMeta fills the reproducibility context that can be read from the binary itself: the Go version and the aikit VCS revision (from the build's embedded VCS info; "" if built without).

type Quality

type Quality struct {
	RecallAtK     *float64 `json:"recall_at_k"`
	ParityOK      bool     `json:"parity_ok"`
	MaxDeltaVsCPU float64  `json:"max_delta_vs_cpu"`
}

Quality couples perf to correctness: a fast-but-wrong row is not admissible. RecallAtK is nil where recall does not apply; ParityOK gates whether the row may be published at all.

type Record

type Record struct {
	Workload       string   `json:"workload"`        // "ann.FlatI8.QueryBatch" | "vision.SigLIP" | ...
	Backend        string   `json:"backend"`         // "cpu-simd" | "metal" | "cuda"
	Device         Device   `json:"device"`          // the hardware this record ran on
	Precision      string   `json:"precision"`       // "f32" | "f16" | "int8" | "int4"
	Shape          Shape    `json:"shape"`           // only the relevant fields are set
	Timing         Timing   `json:"timing_ms"`       // the four cost components + wall, ms
	Throughput     float64  `json:"throughput"`      // items per second (unit below)
	ThroughputUnit string   `json:"throughput_unit"` // "queries/s" | "patches/s" | "tokens/s" | "GB/s"
	Quality        Quality  `json:"quality"`
	SpeedupVsCPU   *float64 `json:"speedup_vs_cpu"` // same box; nil when CPU-only or no CPU on this box
	Meta           Meta     `json:"meta"`
}

Record is one measurement. The field layout mirrors docs/BENCH-gpu.md's schema so records from different machines and repos are comparable and joinable on (workload, shape, precision).

func LoadRecords

func LoadRecords(path string) ([]Record, error)

LoadRecords reads a JSONL file written by AppendRecords. Blank lines are skipped.

type Result

type Result struct {
	Name      string  // "Flat", "HNSW", "FlatI8"
	N, Dim, K int     // corpus size, dimension, top-k
	BuildMs   float64 // index build time
	Recall    float64 // mean recall@k vs the exact Flat top-k (Flat itself = 1.0)
	P50, P95  float64 // query latency percentiles, ms
	P99, Mean float64 // ms
	MemMB     float64 // index storage (HNSW includes the graph, via MarshalBinary)
	QueriesA  int     // number of queries measured

	// AllocsPerQuery and BytesPerQuery come from runtime.MemStats deltas across
	// the measured loop (perf-campaign item 1). They are here because several
	// campaign items moved allocation without moving latency — item 8 removed
	// 12.7 MiB per GTE.Encode for no time change, item 4 cut FlatI8 query
	// allocation 99% for −5% — and a harness that reports only latency cannot
	// see either. Approximate by construction: MemStats is process-wide, so a
	// concurrent GC or another goroutine perturbs it. Read them as a magnitude.
	AllocsPerQuery float64
	BytesPerQuery  float64

	// QPS is throughput under CONCURRENT load: Concurrency goroutines issuing
	// queries at once. It is a different question from 1/Mean — a
	// bandwidth-bound scan saturates well before it runs out of cores (item 16
	// measured Flat.Query plateauing at ~25 GB/s regardless of worker count), so
	// serial latency systematically over-predicts throughput. Zero when the
	// concurrent pass was not run.
	QPS         float64
	Concurrency int
}

Result is one index's measured profile over a query set.

func Run

func Run(corpus, queries [][]float32, k int, cfg ann.Config) []Result

Run benchmarks Flat, HNSW, and FlatI8 over corpus (each vector L2-normalized, the ann invariant), using queries as the workload and k as top-k. Recall is measured against Flat's exact top-k, so Flat reports 1.0 by definition. cfg tunes the HNSW build.

type Shape

type Shape struct {
	N       int `json:"n,omitempty"`       // corpus size
	Dim     int `json:"dim,omitempty"`     // vector dimension
	Batch   int `json:"batch,omitempty"`   // query/batch size
	K       int `json:"k,omitempty"`       // top-k
	Seq     int `json:"seq,omitempty"`     // sequence length
	Patches int `json:"patches,omitempty"` // ViT patch count
}

Shape carries only the fields relevant to a workload (omitempty), so an ANN row and a ViT row share one struct without noise.

type Timing

type Timing struct {
	OneTime   float64 `json:"one_time"`
	PerLaunch float64 `json:"per_launch"`
	H2D       float64 `json:"h2d"`
	D2H       float64 `json:"d2h"`
	Compute   float64 `json:"compute"`
	Wall      float64 `json:"wall"`
}

Timing breaks the cost apart (ms) — never a single fused "GPU time". one_time is amortized (context + pipeline + index residency, measured once); transfer is h2d+d2h and is backend-dependent (explicit copies on CUDA, ~free on Metal UMA); compute is the warm kernel.