Regularizing Neural Machine Translation by Target-bidirectional Agreement
This addresses a fundamental shortcoming in sequence generation for machine translation, though it is an incremental improvement over existing methods.
The paper tackles the problem of error amplification in neural machine translation by proposing a regularization method that improves agreement between left-to-right and right-to-left decoders, resulting in significant performance gains over state-of-the-art baselines on Chinese-English and English-German tasks.
Although Neural Machine Translation (NMT) has achieved remarkable progress in the past several years, most NMT systems still suffer from a fundamental shortcoming as in other sequence generation tasks: errors made early in generation process are fed as inputs to the model and can be quickly amplified, harming subsequent sequence generation. To address this issue, we propose a novel model regularization method for NMT training, which aims to improve the agreement between translations generated by left-to-right (L2R) and right-to-left (R2L) NMT decoders. This goal is achieved by introducing two Kullback-Leibler divergence regularization terms into the NMT training objective to reduce the mismatch between output probabilities of L2R and R2L models. In addition, we also employ a joint training strategy to allow L2R and R2L models to improve each other in an interactive update process. Experimental results show that our proposed method significantly outperforms state-of-the-art baselines on Chinese-English and English-German translation tasks.