KIVI

“Existing static and uniform KV precision methods including KIVI 4-bit cannot effectively handle these non-sparse retrieval heads.”

“Quality degrades sharply below 4 bits.”

“KIVI shows an accuracy drop of 7.89% on LLaMA3-8B model, indicating the suboptimality of preserving recent tokens in full precision instead of identifying salient ones”

“Methods such as KIVI and Kitty maintain a fixed-length residual buffer of unquantized key--value pairs alongside quantized tokens, creating a mixed-precision KV cache. However, PagedAttention manages cache memory in fixed-size, uniform-type blocks; accommodating two distinct precisions within the same paged pool requires either fragmented memory layouts or separate page tables, both of which complicate memory management and break the assumptions of existing fused attention kernels.”

“KIVI~kivi proposes a channel-wise quantization strategy that groups and quantizes key elements along the channel dimensions. However, polar transformation enables smoother distributions of radii and angles, which alleviates the burden of channel-wise quantization outliers.”

“these methods are generally statically configured at runtime: fixed choice of transforms, quantization granularities, and codecs.”

“However, these techniques often achieve modest compression ratios unless combined with additional encoding, which introduces overhead and limits their applicability in latency-sensitive LLM inference.”

“However, further reducing to 2 bits significantly harms model accuracy across a range of downstream tasks.”

“these approaches often experience performance degradation under extreme compression ratios, particularly around 2-bit precision”

“although the sentences generated by KIVI are coherent, the initial words differ from those generated by the original model”

“At the 1-bit quantization, the performance of KIVI and SKVQ has a significant drop.”

“Both apply uniform precision to all tokens within each group, regardless of token importance; uses per-token mixed precision”