HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding
This addresses the need for efficient positional encoding in NLP models, but it is incremental as it builds on existing methods like ALiBi without experimental validation.
The paper tackles the problem of encoding token positions in Transformers by introducing HyPE, a method using hyperbolic functions to bias attention without storing O(L^2) values, and analytically shows it can approximate ALiBi for generalization beyond pretraining lengths.
In Transformer-based architectures, the attention mechanism is inherently permutation-invariant with respect to the input sequence's tokens. To impose sequential order, token positions are typically encoded using a scheme with either fixed or learnable parameters. We introduce Hyperbolic Positional Encoding (HyPE), a novel method that utilizes hyperbolic functions' properties to encode tokens' relative positions. This approach biases the attention mechanism without the necessity of storing the $O(L^2)$ values of the mask, with $L$ being the length of the input sequence. HyPE leverages preliminary concatenation operations and matrix multiplications, facilitating the encoding of relative distances indirectly incorporating biases into the softmax computation. This design ensures compatibility with FlashAttention-2 and supports the gradient backpropagation for any potential learnable parameters within the encoding. We analytically demonstrate that, by careful hyperparameter selection, HyPE can approximate the attention bias of ALiBi, thereby offering promising generalization capabilities for contexts extending beyond the lengths encountered during pretraining. The experimental evaluation of HyPE is proposed as a direction for future research.