Temporal Attention Model for Neural Machine Translation
This work addresses a specific bottleneck in NMT for translation tasks, offering incremental improvements in attention mechanisms.
The paper tackles attention deficiency issues in Neural Machine Translation by introducing a temporal attention mechanism that memorizes alignments within sentences and modulates attention using accumulated temporal memory, achieving better and robust gains over baseline and related NMT approaches in large-scale experiments on two language pairs.
Attention-based Neural Machine Translation (NMT) models suffer from attention deficiency issues as has been observed in recent research. We propose a novel mechanism to address some of these limitations and improve the NMT attention. Specifically, our approach memorizes the alignments temporally (within each sentence) and modulates the attention with the accumulated temporal memory, as the decoder generates the candidate translation. We compare our approach against the baseline NMT model and two other related approaches that address this issue either explicitly or implicitly. Large-scale experiments on two language pairs show that our approach achieves better and robust gains over the baseline and related NMT approaches. Our model further outperforms strong SMT baselines in some settings even without using ensembles.