Learning Hard Retrieval Decoder Attention for Transformers
This work addresses the decoding speed bottleneck in Transformer models for machine translation, offering a practical improvement for real-time applications.
The paper tackled the computational inefficiency of standard multi-head attention in Transformers by introducing a hard retrieval attention mechanism that attends to only one token per head, replacing matrix multiplication with retrieval. This approach achieved a 1.43x speedup in decoding while maintaining translation quality across various machine translation tasks.
The Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. The multi-head attention network performs the scaled dot-product attention function in parallel, empowering the model by jointly attending to information from different representation subspaces at different positions. In this paper, we present an approach to learning a hard retrieval attention where an attention head only attends to one token in the sentence rather than all tokens. The matrix multiplication between attention probabilities and the value sequence in the standard scaled dot-product attention can thus be replaced by a simple and efficient retrieval operation. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding, while preserving translation quality on a wide range of machine translation tasks when used in the decoder self- and cross-attention networks.