LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions
This work addresses a bottleneck in efficient transformers for tasks with unknown sequence lengths, offering a practical improvement for researchers and practitioners in NLP and speech processing.
The paper tackled the limitation of linear transformers in autoregressive and simultaneous tasks by proposing Learned Proportions (LeaP) and LeaPformers, which generalize re-weighting to sequence proportions and use dynamic proportions for flexible attention, achieving the best quality-throughput trade-off on the Long-Range Arena benchmark and competitive results in language modeling and speech-to-text translation.
A promising approach to preserving model performance in linearized transformers is to employ position-based re-weighting functions. However, state-of-the-art re-weighting functions rely heavily on target sequence lengths, making it difficult or impossible to apply them to autoregressive and simultaneous tasks, where the target and sometimes even the input sequence length are unknown. To address this issue, we propose Learned Proportions (LeaP) and LeaPformers. Our contribution is built on two major components. First, we generalize the dependence on explicit positional representations and sequence lengths into dependence on sequence proportions for re-weighting. Second, we replace static positional representations with dynamic proportions derived via a compact module, enabling more flexible attention concentration patterns. We evaluate LeaPformer against eight representative efficient transformers on the Long-Range Arena benchmark, showing that LeaPformer achieves the best quality-throughput trade-off, as well as LeaPformer to Wikitext-103 autoregressive language modeling and simultaneous speech-to-text translation for two language pairs, achieving competitive results.