Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
This addresses a fundamental question in transformer architecture design for researchers, though it is incremental as it builds on known capabilities without positional encodings.
The paper tackles the problem of how causal transformers can store positional information without explicit encodings, finding that nearby embeddings are more similar than faraway ones, enabling position reconstruction in both trained and randomly initialized models across common hyperparameters.
Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.