Universal Conditional Masked Language Pre-training for Neural Machine Translation
This work addresses performance gains in NMT for both autoregressive and non-autoregressive tasks across various language resource levels, representing a novel unified approach rather than an incremental improvement.
The paper tackles the problem of improving Neural Machine Translation (NMT) by pre-training a sequence-to-sequence model with a bidirectional decoder, achieving up to +14.4 BLEU on low-resource languages and +7.9 BLEU on average for autoregressive NMT, and up to +5.3 BLEU for non-autoregressive NMT.
Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT). Different from prior works where pre-trained models usually adopt an unidirectional decoder, this paper demonstrates that pre-training a sequence-to-sequence model but with a bidirectional decoder can produce notable performance gains for both Autoregressive and Non-autoregressive NMT. Specifically, we propose CeMAT, a conditional masked language model pre-trained on large-scale bilingual and monolingual corpora in many languages. We also introduce two simple but effective methods to enhance the CeMAT, aligned code-switching & masking and dynamic dual-masking. We conduct extensive experiments and show that our CeMAT can achieve significant performance improvement for all scenarios from low- to extremely high-resource languages, i.e., up to +14.4 BLEU on low resource and +7.9 BLEU improvements on average for Autoregressive NMT. For Non-autoregressive NMT, we demonstrate it can also produce consistent performance gains, i.e., up to +5.3 BLEU. To the best of our knowledge, this is the first work to pre-train a unified model for fine-tuning on both NMT tasks. Code, data, and pre-trained models are available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/CeMAT.