KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
This addresses the length extrapolation problem for transformer-based language models, representing an incremental improvement over existing relative positional embedding methods.
The paper tackles the problem of length extrapolation in transformer models by proposing KERPLE, a framework that kernelizes relative positional embeddings using conditionally positive definite kernels. Experiments show the logarithmic variant achieves excellent extrapolation performance on three large language modeling datasets.
Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propose KERPLE, a framework that generalizes relative position embedding for extrapolation by kernelizing positional differences. We achieve this goal using conditionally positive definite (CPD) kernels, a class of functions known for generalizing distance metrics. To maintain the inner product interpretation of self-attention, we show that a CPD kernel can be transformed into a PD kernel by adding a constant offset. This offset is implicitly absorbed in the Softmax normalization during self-attention. The diversity of CPD kernels allows us to derive various RPEs that enable length extrapolation in a principled way. Experiments demonstrate that the logarithmic variant achieves excellent extrapolation performance on three large language modeling datasets. Our implementation and pretrained checkpoints are released at https://github.com/chijames/KERPLE.git.