Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
This provides theoretical foundations for improving Transformer models, though it appears incremental as it builds on existing dynamical analysis.
The paper analyzes the dynamical system governing tokens in pre-trained Transformers, characterizing when tokens converge or diverge based on model parameters and showing that convergence harms performance. It proposes simple architectural refinements to mitigate this issue for absolute and rotary positional encodings.
This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of its solutions. Specifically, we characterize when tokens move closer to or farther from one another over time, depending on the model parameters. We provide sufficient conditions, based on these parameters, to identify scenarios where tokens either converge to zero or diverge to infinity. Unlike prior works, our conditions are broader in scope and more applicable to real-world models. Furthermore, we investigate how different forms of positional encoding -- specifically absolute and rotary -- affect these dynamical regimes. Empirical evidence reveals that the convergence scenario adversely impacts model performance. Motivated by these insights, we propose simple refinements to Transformer architectures that mitigate convergence behavior in models with absolute or rotary positional encoding. These findings support theoretical foundations and design principles for improving Transformer models.