Scholar Feed

KV-cache compression

InfiniGen

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

Superseded baseline#15 of 234 most-superseded · first seen Jun 28, 2024

Superseded — cited as a baseline and beaten by newer methods

6 papers critique it · 4 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites InfiniGen as a baseline.

AGX+InfiniGen and AGX+InfiniGenP are even slower than the FlexGen baseline due to fine-grained, token-level selection introducing significant preprocessing overhead.
— V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrieval
“this approach incurs significant overhead due to the latency of fetching the selected sparse KV pairs from the CPU during decoding”
— ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
“However, the estimation time increases significantly as the sequence grows, and the inference time for a single layer is insufficient to cover this.”
— AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
“Prefetching approaches, including FlexGen and InfiniGen, help alleviate PCIe data transfer latency but face limitations in handling large-scale tasks or entail performance trade-offs.”
— CLO: Efficient LLM Inference System with CPU-Light KVCache Offloading via Algorithm-System Co-Design
“Although this approach mitigates the GPU memory capacity constraint, it introduces a new I/O bottleneck. We observe that in InfiniGen, even with prefetching, slow I/O causes the GPU to stall for 61% of the end-to-end execution time, leading to a substantial performance degradation.”
— ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
“InfiniGen's recall latency cannot be fully hidden due to its inefficient token-wise recall.”
— FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference

Beaten on benchmarks

Head-to-head results where a newer method reports beating InfiniGen. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.