Look-ahead Attention for Generation in Neural Machine Translation
This addresses a bottleneck in neural machine translation for improving translation quality by better modeling target-side dependencies, representing an incremental advance.
The paper tackled the problem that neural machine translation models struggle to capture dependencies between distant target words, proposing a look-ahead attention mechanism that achieved substantial improvements over state-of-the-art baselines on Chinese-to-English and English-to-German translation tasks.
The attention model has become a standard component in neural machine translation (NMT) and it guides translation process by selectively focusing on parts of the source sentence when predicting each target word. However, we find that the generation of a target word does not only depend on the source sentence, but also rely heavily on the previous generated target words, especially the distant words which are difficult to model by using recurrent neural networks. To solve this problem, we propose in this paper a novel look-ahead attention mechanism for generation in NMT, which aims at directly capturing the dependency relationship between target words. We further design three patterns to integrate our look-ahead attention into the conventional attention model. Experiments on NIST Chinese-to-English and WMT English-to-German translation tasks show that our proposed look-ahead attention mechanism achieves substantial improvements over state-of-the-art baselines.