H2O

“under unidirectional mask in LLM computations, aggregating attention weights across all query states often causes recent KV cache elements to be mistakenly evicted, degrading the quality of subsequent generations”

“However, these methods depend on attention scores for eviction, requiring CUDA kernel modifications to track them.”

“the majority of studies adopt cumulative attention scores as the criterion for token pruning”

“While such techniques effectively reduce the pressure on memory bandwidth during the attention computation, they typically do not reduce the physical storage requirements of the KV cache; the full context remains resident in memory, even if only a fraction is accessed during decode.”

“Existing approaches use heuristics like discarding oldest tokens~fastgen, streamingllm or leverage attention scores from past queries~h2o, snapkv, tova, but these strategies are limited for real-world scenarios”

“these methods rely on predefined retention rules and cannot adapt to evolving inference dynamics”

“Although these methods generally have low additional overhead, they often lead to noticeable performance degradation.”

“Methods such as SnapKV li2024snapkv and H2O zhang2024h2o apply this strategy to vision-language modeling (VLM) tasks by treating visual and text tokens uniformly across long sequences during pruning. Unfortunately, these methods rely on original attention scores that mix different modalities, potentially leading to suboptimal pruning outcomes.”

“these methods either (i) optimize throughput while leaving allocation semantics untouched or (ii) assume a monolithic, forward-moving path, failing to model the topological constraints and frequent backtracking inherent in tree-structured search”

“post-hoc compression algorithms usually evict KV pairs based on attention scores, which is not compatible with FlashAttention and thus prevents their applications in modern LLMs inference systems.”

“H2O and StreamingLLM allocate the same cache budget across all layers, causing denser layers to miss important tokens in the context, while sparser layers contain redundant tokens.”

“However, discarding tokens permanently erases their information, which proves to be suboptimal for tasks such as retrieval”