Improving Transformers using Faithful Positional Encoding
This work addresses a specific bottleneck in Transformer architectures for researchers and practitioners in sequence modeling, though it appears incremental as it builds upon existing positional encoding methods.
The authors tackled the problem of positional encoding in Transformers by introducing a mathematically grounded method that guarantees no loss of positional order information, resulting in systematic performance improvements in time-series classification tasks.
We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.