q8kernel

Documentation

Overview

Command q8kernel is a self-contained kernel microbenchmark that isolates ONE question from all model/WSL/load overhead: in pure Go on this box, which GEMV kernel is fastest for the memory-bound batch-1 decode regime —

f32        : y[o] = Σ w[o,i]·x[i]            (the current f32 path; 4 B/weight, vectorized FMA)
int8×int8  : y[o] = Σ blocks dw·dx·Σ(qw·qx)  (the shipped Q8_0 path; 1 B/weight, scalar int dot)
int8×f32   : y[o] = s[o]·Σ float32(qw[i])·x[i] (weight-only; 1 B/weight, f32 mul, NO activation quant)

It streams a head-sized weight matrix (49152×576 ≈ 113 MB f32 / 28 MB int8 — far past L3, so it is genuinely DRAM-bandwidth-bound like real decode), row-parallel across all cores, and reports ms/matmul and effective weight-GB/s. This answers whether descending to int8 can beat f32 in pure Go WITHOUT touching the model package (no collision), and whether the bandwidth win (4× fewer bytes) survives the per-element conversion cost.