Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translation
This addresses the challenge of enhancing translation accuracy for multilingual systems, especially in low-resource and zero-shot scenarios, with incremental improvements over existing methods.
The paper tackles the problem of improving direct translation performance in multilingual models, particularly in zero-shot settings, by modifying input tokens to include source and target language signals, resulting in gains of up to 10.0 BLEU points on in-house datasets and 4.17 BLEU points in WMT evaluations.
This paper proposes a simple yet effective method to improve direct (X-to-Y) translation for both cases: zero-shot and when direct data is available. We modify the input tokens at both the encoder and decoder to include signals for the source and target languages. We show a performance gain when training from scratch, or finetuning a pretrained model with the proposed setup. In the experiments, our method shows nearly 10.0 BLEU points gain on in-house datasets depending on the checkpoint selection criteria. In a WMT evaluation campaign, From-English performance improves by 4.17 and 2.87 BLEU points, in the zero-shot setting, and when direct data is available for training, respectively. While X-to-Y improves by 1.29 BLEU over the zero-shot baseline, and 0.44 over the many-to-many baseline. In the low-resource setting, we see a 1.5~1.7 point improvement when finetuning on X-to-Y domain data.