A Simple and Effective Positional Encoding for Transformers
This work addresses the need for more efficient positional encodings in Transformers, offering a simple solution that improves speed and maintains performance, though it is incremental relative to existing methods.
The authors tackled the problem of efficiently encoding positional and segment information in Transformer models by introducing Decoupled Positional Attention for Transformers (DIET), which moves positional information to the attention layer, resulting in faster training and inference times while achieving competitive performance on benchmarks like GLUE, XTREME, and WMT.
Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.