Scholar Feed

KV-cache compression

KVQuant

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

Superseded baseline#10 of 234 most-superseded · first seen Jan 31, 2024

Superseded — cited as a baseline and beaten by newer methods

6 papers critique it · 6 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites KVQuant as a baseline.

To mitigate this, KVQuant~kvquant proposes quantizing the keys before applying RoPE, which is described as pre-RoPE quantization. Promising as it is, this approach requires on-the-fly RoPE computation, which consequently introduces potential computational overhead.
— PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration
“these methods are generally statically configured at runtime: fixed choice of transforms, quantization granularities, and codecs.”
— KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
“RTN, SKVQ, and KVQuant exhibit significant performance degradation at ultra-low bit-widths.”
— AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
“Both apply uniform precision to all tokens within each group, regardless of token importance; uses per-token mixed precision”
— SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving
“KIVI/KVQuant primarily target two bits or above”
— FibQuant: Universal Vector Quantization for Random-Access KV-Cache Compression
“It uses a calibration dataset to compute the fisher matrix and find the signposts before inference begins.”
— InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models

Beaten on benchmarks

Head-to-head results where a newer method reports beating KVQuant. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.