Alternative positional encoding functions for neural transformers
This work addresses a key module in transformer architectures, potentially improving performance for various AI applications, though it appears incremental.
The authors tackled the problem of positional encoding in neural transformers by proposing an alternative set of periodic functions, which substantially outperformed the original sinusoidal version in tentative experiments.
A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted in the use of sinusoidal functions of various frequencies, in order to capture recurrent patterns of differing typical periods. In this work, an alternative set of periodic functions is proposed for positional encoding. These functions preserve some key properties of sinusoidal ones, while they depart from them in fundamental ways. Some tentative experiments are reported, where the original sinusoidal version is substantially outperformed. This strongly suggests that the alternative functions may have a wider use in other transformer architectures.