On the use of BERT for Neural Machine Translation
This work addresses translation robustness for NMT practitioners by exploring BERT integration, but it is incremental as it builds on existing pretrained model methods.
The paper investigates integrating BERT pretrained models into supervised neural machine translation (NMT) to improve translation quality, finding that it enhances performance on standard and out-of-domain test sets, with specific gains reported on datasets like WMT-14 English-German.
Exploiting large pretrained models for various NMT tasks have gained a lot of visibility recently. In this work we study how BERT pretrained models could be exploited for supervised Neural Machine Translation. We compare various ways to integrate pretrained BERT model with NMT model and study the impact of the monolingual data used for BERT training on the final translation quality. We use WMT-14 English-German, IWSLT15 English-German and IWSLT14 English-Russian datasets for these experiments. In addition to standard task test set evaluation, we perform evaluation on out-of-domain test sets and noise injected test sets, in order to assess how BERT pretrained representations affect model robustness.