KV-cache compression

SnapKV: LLM Knows What You are Looking for Before Generation

51 critique · 71 beaten on benchmarks

H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

65 critique · 56 beaten on benchmarks

Efficient Streaming Language Models with Attention Sinks

43 critique · 44 beaten on benchmarks

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

21 critique · 29 beaten on benchmarks

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

20 critique · 27 beaten on benchmarks

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

13 critique · 16 beaten on benchmarks

Transformers are Multi-State RNNs

6 critique · 14 beaten on benchmarks

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

7 critique · 7 beaten on benchmarks

6 critique · 6 beaten on benchmarks

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

6 critique · 6 beaten on benchmarks

TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate

7 critique · 4 beaten on benchmarks

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

8 critique · 3 beaten on benchmarks