Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models
This addresses the challenge for industries needing domain-specific translation with limited data, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of adapting pre-trained multilingual neural machine translation models to specialized domains without degrading performance on generic data across all language pairs, achieving a +10.0 BLEU score gain on specialized data with minimal losses of -0.01 to -0.5 BLEU on generic datasets.
Recent literature has demonstrated the potential of multilingual Neural Machine Translation (mNMT) models. However, the most efficient models are not well suited to specialized industries. In these cases, internal data is scarce and expensive to find in all language pairs. Therefore, fine-tuning a mNMT model on a specialized domain is hard. In this context, we decided to focus on a new task: Domain Adaptation of a pre-trained mNMT model on a single pair of language while trying to maintain model quality on generic domain data for all language pairs. The risk of loss on generic domain and on other pairs is high. This task is key for mNMT model adoption in the industry and is at the border of many others. We propose a fine-tuning procedure for the generic mNMT that combines embeddings freezing and adversarial loss. Our experiments demonstrated that the procedure improves performances on specialized data with a minimal loss in initial performances on generic domain for all languages pairs, compared to a naive standard approach (+10.0 BLEU score on specialized data, -0.01 to -0.5 BLEU on WMT and Tatoeba datasets on the other pairs with M2M100).