CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
This work addresses a key limitation in Transformer models for researchers and practitioners, offering a more robust and efficient solution for encoding positional information, though it is incremental as it builds on existing absolute and relative embedding methods.
The paper tackles the problem of positional embeddings in Transformers by proposing CAPE, an augmentation-based approach that combines the simplicity and speed of absolute embeddings with the better generalization of relative embeddings, leading to improved generalization performance and training stability across machine translation, image, and speech recognition tasks.
Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters.