Scholar Feed

KV-cache compression

InfLLM

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

Superseded baseline#18 of 234 most-superseded · first seen Feb 7, 2024

Superseded — cited as a baseline and beaten by newer methods

5 papers critique it · 4 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites InfLLM as a baseline.

However, due to its sub-optimal block-level selection, it results in lower performance on most tasks compared to TokenSelect, even though we set a larger token budget for InfLLM.
— TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
“They require careful hyperparameter tuning (e.g., chunk size in InfLLM~xiao2024infllm, or ANN index construction in RetrievalAttention~liu2024retrievalattention) and must retain the full KV cache as a candidate pool, limiting memory savings.”
— LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
“this increases latency because of the newly introduced retrieval overhead which was not present in legacy methods”
— More Than a Quick Glance: Overcoming the Greedy Bias in KV-Cache Compression
“Though the block-level space-continuity assumption improves efficiency, it does not align with real scenarios where relevant tokens are distributed discretely, leading to a significant drop in model quality.”
— PQCache: Product Quantization-based KVCache for Long Context LLM Inference
“Although CPU offloading mitigates GPU memory limitations, existing approaches~xiao2024infllm,zhang2024pqcache still require retrieving a substantial portion of tokens (around 20\%), introducing significant decoding latency overheads due to slow data transfer between CPU RAM and GPU RAM.”
— TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

Beaten on benchmarks

Head-to-head results where a newer method reports beating InfLLM. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.