SHAPE: Shifted Absolute Position Embedding for Transformers
This addresses a domain-specific issue for researchers and practitioners using Transformers in NLP or other sequence tasks, but it is incremental as it builds on existing position representation methods.
The paper tackled the problem of position representation in Transformers, which often lacks generalization to unseen lengths or incurs high computational costs, by proposing SHAPE (Shifted Absolute Position Embedding) that uses random shifting during training to achieve shift invariance, resulting in a method that is empirically comparable to existing approaches while being simpler and faster.
Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.