AdaKV

“However, these methods often rely on experimental observations and pre-define some rules for cache budget allocation and KV Cache eviction.”

“proposed dynamic head-level allocation using attention scores but still relied on layer-level budgeting.”

“AdaKV~adakv achieves theoretical optimality in allocation at the attention score level given a fixed budget, though this does not always translate to optimal end-to-end performance. These works make important contributions to the allocation problem, but they also exacerbate the fundamental challenge: how should the budget be determined in the first place?”

“adaptive methods like Ada-KV and D2O rely on calculating attention scores to allocate budgets or select tokens. This creates an inference-time circular dependency: identifying important components requires performing the heavy query-key interactions (O(t^2) complexity) that we aim to avoid”

“existing methods evaluate attention head importance independently. For example, AdaKV evaluates the concentration degrees of heads while HeadKV assesses the retrieval-reasoning capability of each head in isolation as a measure of importance. However, these approaches treat heads as isolated units, overlooking the fact that their true importance emerges from their cooperation rather than individual capabilities.”

“all eviction methods share the same post-eviction inference procedure: attention is renormalized exclusively over the retained KV pairs, and the evicted ones leave no trace in subsequent operations”