Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
This addresses the problem of simplifying transformer architectures for researchers and practitioners by potentially eliminating the need for positional embeddings, though it appears incremental as it builds on prior questioning of their necessity.
The paper demonstrates that transformer language models without positional embeddings inherently encode positional information through self-attention variance shrinkage, showing this effect persists after pretraining, which could enable more efficient model training.
The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.