Documentation
¶
Overview ¶
Package quant turns weights into the form a quantized kernel reads.
A quantized weight matrix is two arrays rather than one: the quants, one small integer per weight, and the scales, one per block of weights. You get both from [Int8], hand them to the device as an I8 buffer and an F16 buffer, and a quantized operator reads them together.
q, s := quant.Int8Quantize(weights)
What it costs you ¶
Accuracy, by a bounded amount. A weight comes back as Int8Block values times its block's scale, which is within half a scale step of what you put in — so the error is proportional to the *largest* weight in each block, and a block containing one outlier represents its other 31 weights worse than a block without one. [Error] computes the bound for a dot product so a test can assert against it rather than against a tolerance somebody tuned.
What it saves you ¶
Half the memory of f16 and a quarter of f32, and on a memory-bound decode step that is most of the run time.
specs/027-quantization.md has the derivation and the reasoning.
Index ¶
- Constants
- func Int4Dequantize(packed []uint32, scales, zeros []accel.Float16, n int) []float32
- func Int4ErrorBound(x []float32, termRanges []float32) float64
- func Int4Nibble(packed []uint32, i int) uint32
- func Int4Quantize(w []float32) (packed []uint32, scales, zeros []accel.Float16)
- func Int8Dequantize(quants []int8, scales []accel.Float16) []float32
- func Int8ErrorBound(x []float32, termScales []accel.Float16) float64
- func Int8Quantize(w []float32) (quants []int8, scales []accel.Float16)
Constants ¶
const Int4Group = 128
Int4Group is how many weights share one scale and one zero point.
A hundred and twenty-eight, and the reason is not Int8Block's. That is 32 because the tiled GEMM steps K in sixteen -- a tiling choice, made when the metadata it implied was invisible. At four bits it is not: halving the payload doubles the metadata's share of it, so an fp16 scale per 32 costs 12.5% of a 4-bit format where it costs 6.2% of an 8-bit one.
A group of 128 carries *twice* as much -- a scale and a zero -- and still costs 6.2%, because it is amortised over four times as many weights. And 128 is a multiple of sixteen and is exactly the row kernels' width, so the argument that fixed 32 permits this rather than forbidding it.
It is also what AWQ and GPTQ publish, which is what lets one representation read the ecosystem's checkpoints and give a caller a format to quantize into.
const Int4Max = 15
Int4Max is the largest code a 4-bit weight takes.
Fifteen, and the range is [0, 15] rather than symmetric about zero: see Int4Quantize for why asymmetry is the whole design at this width.
const Int8Block = 32
Int8Block is how many weights share one scale.
Thirty-two, because specs/010-kernel-corpus.md's tiled GEMM steps K in sixteen and a multiple of that means no K-step ever straddles a scale boundary: a step reads one scale rather than two. The row kernels are 128 wide, also a multiple.
const Int8Max = 127
Int8Max is the largest magnitude a quant takes.
127 rather than 128, so the representable range is symmetric about zero. int8 reaches -128, and using it would make -128*s a value with no positive counterpart -- which is exactly the special case the error bound in specs/027-quantization.md is stated without.
Variables ¶
This section is empty.
Functions ¶
func Int4Dequantize ¶
Int4Dequantize reconstructs the weights a packed group represents.
n is the weight count rather than derived from len(packed), because a count that is not a multiple of eight leaves the last word's high nibbles holding codes nobody wrote.
func Int4ErrorBound ¶
Int4ErrorBound is the quantization term of a dot product, over the inputs it used.
specs/048-int4.md §3. Derived, not observed: rounding to nearest gives at most half a step, and the step is a group's *range* over fifteen where Int8ErrorBound's is a peak over 127.
|sum (q-z)s x - sum w x| <= (1/30) sum |x_i| (max_g(i) - min_g(i))
termRanges is one group range per term, for Int8ErrorBound's reason: a per-group figure bounds a dot product only where the caller says which group each term came from.
func Int4Nibble ¶
Int4Nibble reads the 4-bit code at index i.
Exported because it is the one piece of this representation a kernel and a caller must spell identically, and so the place where two implementations disagree without either being obviously wrong.
func Int4Quantize ¶
Int4Quantize packs weights into 4-bit codes with a scale and zero per group.
Why asymmetric, where int8 is symmetric ¶
Int8Quantize centres its range on zero, which is right at eight bits. At four there are sixteen codes, and spending them symmetrically means a group whose weights all sit near one nonzero value wastes almost all of them. So a group carries a scale *and* a zero point:
s = (max - min) / 15, z = -min / s, w ~= (q - z) * s
specs/048-int4.md §3 states what that buys and costs: the error is a *range* over 30 where int8's is a *peak* over 254, so a group clustered away from zero is represented better by four asymmetric bits than by eight symmetric ones, and a group centred on zero is about seventeen times worse.
The packing is into words, not bytes ¶
Eight weights per uint32, low nibble first: word j holds weights 8j..8j+7 with weight 8j+n in bits 4n..4n+3. A group of 128 is 16 words, so a group boundary is always a word boundary and no weight straddles one.
Words rather than bytes because a kernel cannot do arithmetic on a u8 at all: specs/002-compute-model.md makes narrow dtypes *storage*, converted to f32 on load, so a shift and a mask on one is outside the subset. Byte packing would have needed the caller to reinterpret the slice as words on upload, which works only because every supported platform is little-endian -- a dependency worth not having when the alternative removes it. It is also what AWQ and GPTQ pack into, so the ecosystem's files need no repacking either.
The returned slice is ceil(len(w)/8) words; at a length that is not a multiple of eight the last word's high nibbles are zero and read back as weights nobody asked for, which is why Int4Dequantize takes the count rather than deriving it.
func Int8Dequantize ¶
Dequantize reconstructs the weights a quantized pair represents.
The reference for what a kernel must compute, and the thing to compare against when asking how much accuracy a quantization cost.
func Int8ErrorBound ¶
Error bounds how far a quantized dot product may sit from the exact one.
Why a bound rather than a tolerance ¶
specs/008-numerics.md admits no tolerance parameter anywhere, and this is why: a tolerance is a number somebody raised until a test passed. This is derived from the representation. Rounding to nearest puts each weight within half a scale step of its original, a step is the block's largest magnitude over 127, and a dot product weights each of those errors by the activation it multiplies:
|Σ qᵢsᵢxᵢ − Σ wᵢxᵢ| ≤ (1/254) Σ |xᵢ| · max|w in block(i)|
It is an absolute bound over the actual inputs, so a test computes it from the values it used rather than guessing a relative one.
**This covers quantization only.** The products are summed in f32, so specs/008-numerics.md section 7's reduction bound applies to the sum on top, and a caller comparing against an exact reference adds the two. It takes one scale **per term**, not the quantized array's per-block scales.
That is the whole correction. It used to index scales[i/Int8Block], which is a bound only when the dot product's terms are contiguous in the quantized array — true for a row, false for a column. A caller who passed a column's activations and the matrix's scales got a plausible number that was not a bound, and this repository already worked around it by building a per-term array and reimplementing the loop inline. Now the signature asks for what the arithmetic needs, so the workaround is the call.
func Int8Quantize ¶
Int8 quantizes weights into per-block scaled integers.
The returned quants are one per weight and the scales one per Int8Block of them, padded when the length is not a multiple: a trailing partial block gets its own scale over the weights it has.
Types ¶
This section is empty.
Source Files