PyramidKV

“existing methods often set an attenuation coefficient to control the KV cache budget in each layer, thus ignoring that the cache budget actual needed in each layer do not necessarily exhibit a monotonically decreasing pattern.”

“PyramidKV statically allocates KV cache in a monotonically decreasing manner, which is ineffective for all input queries.”

“Figure fig:score_heatmap reveals significant differences between individual layers even within the conventionally grouped Shallow (Layer ID 0-9), Middle (10-19), and Deep (20-31) layers, thereby challenging existing three-part perspectives in PyramidKV”

“While these methods differ in selecting tokens for KV cache retention, they generally apply a uniform budget size across layers, even though the optimal budget size may vary.”

“Hierarchical methods like PyramidKV zhang2024pyramidkv adapt by layer but lack generalizability.”

“However they still quantize (if at all) uniformly.”

“However, these methods often rely on experimental observations and pre-define some rules for cache budget allocation and KV Cache eviction.”

“These methods make binary keep/drop decisions per token; retains the importance idea but replaces the drop action with lower-precision transmission”

“Our work shares the insight that uniform allocation is suboptimal, but differs in signal (dynamic pilot MSE vs. static attention patterns) and scope (we also allocate across KV heads within a layer).”

“Despite their success in reducing cache size, these methods predominantly rely on static importance scores, overlooking the dynamic, implicit relationships among tokens.”

“Although more effective than uniform allocation, this approach relies on heuristic allocation rather than learned patterns.”

“For Prefill-Only Compression, methods like SnapKV and PyramidKV, retaining all KV cache generated during the decoding phase, leading to linear cache growth with the output length and memory pressure.”