TurboQuant

“per-vector methods treat each vector independently, ignoring the structured nature of the KV cache. Within an attention head, a block of n consecutive key or value vectors is not a collection of independent random vectors—it contains a low-rank component reflecting shared structure across tokens. This shared structure means the quantizer's theoretical assumptions (isotropy on the unit sphere) are not fully satisfied, leading to inner product bias.”

“The main drawback is cost. For a head dimension d, a dense orthogonal transform requires O(d²) parameters and arithmetic, which is difficult to justify in latency-sensitive settings such as autoregressive decoding.”

“But it is the wrong geometry. Once a vector has been normalized and Haar-rotated, a block of k consecutive coordinates lies on the unit ball with a specific radial law and a uniform angular component. The coordinates are not an independent product of shifted-Beta marginals. A scalar code sees one coordinate at a time; the source seen by the cache is intrinsically vectorial.”

“Based on our experimental results, InnerQ achieves a comparable evaluation score to TurboQuant (Section~sec:accuracy) while having a lower latency (Section~sec:speedup).”

“TurboQuant's lower bound is tight---for the problem it solves. That problem is: given an isolated KV vector drawn from the post-rotation distribution, what is the minimum number of bits needed to represent it? The paper's answer is approximately 3 bits per component, and TurboQuant achieves it. But the KV cache is not a collection of isolated vectors.”

“On Mistral-7B, TurboAngle with $n = 64$ (3.0 angle bits) achieves $ = {+}0.0010$, while TurboQuant sym4-g4 at 4.0 bits degrades by ${+}0.0148$: $14.8$ more distortion at a higher bit rate.”

“All three quantize one coordinate (or one angle) at a time.”