Fully Synthetic Data Improves Neural Machine Translation with Knowledge Distillation
This work provides an incremental improvement for neural machine translation practitioners seeking to leverage monolingual data more effectively for knowledge distillation.
This paper investigates using synthetic data to improve neural machine translation through knowledge distillation. They found that round-trip translating target language monolingual text to create a fully synthetic corpus, when combined with forward-translated source language monolingual data, outperforms a corpus twice as large in a single language.
This paper explores augmenting monolingual data for knowledge distillation in neural machine translation. Source language monolingual text can be incorporated as a forward translation. Interestingly, we find the best way to incorporate target language monolingual text is to translate it to the source language and round-trip translate it back to the target language, resulting in a fully synthetic corpus. We find that combining monolingual data from both source and target languages yields better performance than a corpus twice as large only in one language. Moreover, experiments reveal that the improvement depends upon the provenance of the test set. If the test set was originally in the source language (with the target side written by translators), then forward translating source monolingual data matters. If the test set was originally in the target language (with the source written by translators), then incorporating target monolingual data matters.