SnapKV

“SnapKV~snapkv further optimizes this by identifying key information clusters within an attention window.”

“these methods require access to the full attention matrix, making them incompatible with Flash Attention~flashattention and thus impractical for modern deployment scenarios”

“these methods rely on predefined retention rules and cannot adapt to evolving inference dynamics”

“Although these methods generally have low additional overhead, they often lead to noticeable performance degradation.”

“Methods such as SnapKV li2024snapkv and H2O zhang2024h2o apply this strategy to vision-language modeling (VLM) tasks by treating visual and text tokens uniformly across long sequences during pruning. Unfortunately, these methods rely on original attention scores that mix different modalities, potentially leading to suboptimal pruning outcomes.”

“post-hoc compression algorithms usually evict KV pairs based on attention scores, which is not compatible with FlashAttention and thus prevents their applications in modern LLMs inference systems.”

“SnapKV, the current state-of-the-art, achieves high accuracy in long-context tasks by retaining the most attended tokens from the input prompt. However, it retains all generated output tokens from the decode phase, causing the KV cache size to scale with response lengths, making it unsuitable for long-response tasks.”

“While effective for long contexts, they underperform in long generations because discarded KV cannot be reused even if it later becomes important.”

“While these methods differ in selecting tokens for KV cache retention, they generally apply a uniform budget size across layers, even though the optimal budget size may vary.”

“these methods provide at best marginal accuracy improvements over SnapKV, while leaving the fundamental bottleneck unsolved: they still require producing the KV cache for the full-context before selecting which tokens to retain, so prefill latency remains unreduced.”

“eviction-based approaches rely heavily on current token importance assessments. This risks unintentionally and permanently discarding tokens essential for subsequent decoding steps, leading to contextual degradation.”

“Despite their differences, these methods all apply a single eviction policy to every layer.”