A Comparative Analysis of Contextual Representation Flow in State-Space and Transformer Architectures
This work addresses a fundamental gap in understanding representation propagation in efficient long-sequence architectures, providing insights for researchers and practitioners in machine learning, though it is incremental as it builds on existing models without introducing new paradigms.
The paper tackled the problem of understanding how contextual information flows in State Space Models (SSMs) and Transformer-Based Models (TBMs) for long-sequence processing, finding that TBMs rapidly homogenize token representations early with diversity reemerging later, while SSMs preserve token uniqueness early but converge to homogenization deeper.
State Space Models (SSMs) have recently emerged as efficient alternatives to Transformer-Based Models (TBMs) for long-sequence processing, offering linear scaling and lower memory use. Yet, how contextual information flows across layers and tokens in these architectures remains understudied. We present the first unified, token- and layer-level analysis of representation propagation in SSMs and TBMs. Using centered kernel alignment, stability metrics, and probing, we characterize how representations evolve within and across layers. We find a key divergence: TBMs rapidly homogenize token representations, with diversity reemerging only in later layers, while SSMs preserve token uniqueness early but converge to homogenization deeper. Theoretical analysis and parameter randomization further reveal that oversmoothing in TBMs stems from architectural design, whereas in SSMs it arises mainly from training dynamics. These insights clarify the inductive biases of both architectures and inform future model and training designs for long-context reasoning.