Documentation
¶
Overview ¶
Command batchbench measures the AGGREGATE decode throughput of the multi-user batched decode (internal/model.BatchSession) as a function of batch size B — the "continuous batching" / multi-user serving regime MODEL-BASELINE-RESULTS.md scoped out as "vLLM's claim, not fak's". This is the throughput lane: batch-1 decode is memory-bandwidth-bound (the weights are re-streamed per token), so stacking B users' decode steps into one GEMM per layer amortises that weight stream B-fold and aggregate tokens/sec scales with B until the GEMM goes compute-bound.
The headline number is the THROUGHPUT MULTIPLIER: aggregate tok/s at the best batch size divided by the batch-1 (serial-equivalent) tok/s — and, for the cumulative story, divided by the naive f32-serial baseline (52.1 ms/tok = 19.2 tok/s) the whole optimisation effort started from in Act 1.
Apples-to-apples: every B runs the SAME per-user work (one short prompt prefill + D decode steps). Token VALUES never affect matmul/attention cost, so deterministic LCG ids measure the identical work a real fleet of B concurrent users would drive. Pin FAK_WORKERS to fix the core budget; default uses GOMAXPROCS.
Source Files