Eigen Attention

“EigenAttention~saxena2024eigen and Zack~zhang2024zack attempt to address this by incorporating both queries and keys in low-rank decompositions, yet their behavior largely resembles that of SVD-based methods that compress keys alone.”

“Others, like Eigen Attention saxena2024eigenattentionattentionlowrank, tackle the memory issue but require modifying model weights offline for a fixed compression level, sacrificing crucial runtime flexibility.”

“A key limitation of EigenAttention and MatryoshkaKV lies in their use of a static basis: once computed, the projections remain fixed throughout inference. This assumption breaks down when inference prompts diverge from the calibration distribution (e.g., shifting from conversational text to code), leading to degraded approximation and reduced generation quality.”

“Eigen Attention~saxena2024eigen has been proposed to compress the KV cache after applying RoPE but suffers a relatively large accuracy loss.”

“they still optimize pre-softmax or intermediate proxies rather than the full decoder-layer output”