Beyond Position: the emergence of wavelet-like properties in Transformers
This addresses the problem of positional encoding constraints in Transformers for researchers, revealing an incremental insight into model behavior rather than a new paradigm.
The paper investigates how Transformers with Rotary Position Embeddings (RoPE) develop emergent wavelet-like properties to compensate for positional encoding limitations, showing that attention heads evolve to implement multi-resolution processing analogous to wavelet transforms, which emerges through distinct evolutionary phases during training and adheres to the uncertainty principle.
This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical limitations. Through an analysis spanning model scales, architectures, and training checkpoints, we show that attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. We demonstrate that this scale-invariant behavior is unique to RoPE, emerges through distinct evolutionary phases during training, and statistically adheres to the fundamental uncertainty principle. Our findings suggest that the effectiveness of modern Transformers stems from their remarkable ability to spontaneously develop optimal, multi-resolution decompositions to address inherent architectural constraints.