Dynamic Fusion: Attentional Language Model for Neural Machine Translation
This work addresses the challenge of improving translation fluency and adequacy in NMT for language pairs like English-Japanese, though it appears incremental as it builds on existing two-model approaches.
The paper tackles the problem of incorporating language models into neural machine translation by proposing Dynamic Fusion, an attentive architecture that adaptively weights the language model based on translation history, resulting in improved BLEU and RIBES scores for English-Japanese translation.
Neural Machine Translation (NMT) can be used to generate fluent output. As such, language models have been investigated for incorporation with NMT. In prior investigations, two models have been used: a translation model and a language model. The translation model's predictions are weighted by the language model with a hand-crafted ratio in advance. However, these approaches fail to adopt the language model weighting with regard to the translation history. In another line of approach, language model prediction is incorporated into the translation model by jointly considering source and target information. However, this line of approach is limited because it largely ignores the adequacy of the translation output. Accordingly, this work employs two mechanisms, the translation model and the language model, with an attentive architecture to the language model as an auxiliary element of the translation model. Compared with previous work in English--Japanese machine translation using a language model, the experimental results obtained with the proposed Dynamic Fusion mechanism improve BLEU and Rank-based Intuitive Bilingual Evaluation Scores (RIBES) scores. Additionally, in the analyses of the attention and predictivity of the language model, the Dynamic Fusion mechanism allows predictive language modeling that conforms to the appropriate grammatical structure.