Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis
This work addresses the challenge of training efficient language models on short sequences that can generalize to longer ones, offering a novel, incremental improvement in positional embedding design for NLP researchers.
The paper tackles the problem of transformer length extrapolation by analyzing the receptive field of ALiBi and introducing Sandwich, a parameter-free relative positional embedding that enables models to handle sequences longer than those seen in training, achieving competitive perplexities on extended sequences.
Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create ~\textbf{Sandwich}, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.