State Rank Dynamics in Linear Attention LLMs
This work addresses the opacity of state dynamics in Linear Attention LLMs, offering a method to reduce computational overhead for efficient inference, though it is incremental as it builds on existing linear attention models.
The study investigated the internal state dynamics of Linear Attention LLMs, uncovering a phenomenon called State Rank Stratification where heads split into low-rank and high-rank groups, with low-rank heads being crucial for reasoning and high-rank heads redundant, leading to a proposed pruning strategy that reduces KV-cache overhead by 38.9% while maintaining accuracy.
Linear Attention Large Language Models (LLMs) offer a compelling recurrent formulation that compresses context into a fixed-size state matrix, enabling constant-time inference. However, the internal dynamics of this compressed state remain largely opaque. In this work, we present a comprehensive study on the runtime state dynamics of state-of-the-art Linear Attention models. We uncover a fundamental phenomenon termed State Rank Stratification, characterized by a distinct spectral bifurcation among linear attention heads: while one group maintains an effective rank oscillating near zero, the other exhibits rapid growth that converges to an upper bound. Extensive experiments across diverse inference contexts reveal that these dynamics remain strikingly consistent, indicating that the identity of a head,whether low-rank or high-rank,is an intrinsic structural property acquired during pre-training, rather than a transient state dependent on the input data. Furthermore, our diagnostic probes reveal a surprising functional divergence: low-rank heads are indispensable for model reasoning, whereas high-rank heads exhibit significant redundancy. Leveraging this insight, we propose Joint Rank-Norm Pruning, a zero-shot strategy that achieves a 38.9\% reduction in KV-cache overhead while largely maintaining model accuracy.