KV-cache compression

Most-superseded baselines

Ranked by how many distinct papers critique or beat each method. These are the standard baselines newer work routinely measures against.

1. 1

   SnapKV· SnapKV

   SnapKV: LLM Knows What You are Looking for Before Generation

   51 papers critique it · 71 beat it on benchmarks

2. 2

   H2O· SnapKV

   H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

   65 papers critique it · 56 beat it on benchmarks

3. 3

   StreamingLLM· SnapKV

   Efficient Streaming Language Models with Attention Sinks

   43 papers critique it · 44 beat it on benchmarks

4. 4

   PyramidKV· SnapKV

   PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

   21 papers critique it · 29 beat it on benchmarks

5. 5

   KIVI· KIVI

   KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

   20 papers critique it · 27 beat it on benchmarks

6. 6

   Quest· Quest

   Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

   13 papers critique it · 16 beat it on benchmarks

7. 7

   TOVA· SnapKV

   Transformers are Multi-State RNNs

   6 papers critique it · 14 beat it on benchmarks

8. 8

   AdaKV· SnapKV

   Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

   7 papers critique it · 7 beat it on benchmarks

9. 9

   CaM· SnapKV

   6 papers critique it · 6 beat it on benchmarks

10. 10

    KVQuant· KIVI

    KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

    6 papers critique it · 6 beat it on benchmarks

11. 11

    TurboQuant· KIVI

    TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate

    7 papers critique it · 4 beat it on benchmarks

12. 12

    Scissorhands· SnapKV

    Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

    8 papers critique it · 3 beat it on benchmarks