PaTH Attention: Position Encoding via Accumulating Householder Transformations
This addresses a key bottleneck in large language models by improving position encoding for better sequence modeling, though it appears incremental as an enhancement over RoPE.
The paper tackles the limited expressivity of Rotary Position Encoding (RoPE) in transformers by introducing PaTH, a data-dependent position encoding scheme based on Householder transformations, which shows superior performance in synthetic benchmarks and real-world language modeling experiments.
The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is essential for modeling structured domains such as language. Rotary position encoding (RoPE) has emerged as the de facto standard approach for position encoding and is part of many modern LLMs. However, in RoPE the key/query transformation between two elements in a sequence is only a function of their relative position and otherwise independent of the actual input. This limits the expressivity of RoPE-based transformers. This paper describes PaTH, a flexible data-dependent position encoding scheme based on accumulated products of Householder(like) transformations, where each transformation is data-dependent, i.e., a function of the input. We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices, and implement a FlashAttention-style blockwise algorithm that minimizes I/O cost. Across both targeted synthetic benchmarks and moderate-scale real-world language modeling experiments, we find that PaTH demonstrates superior performance compared to RoPE and other recent baselines.