Cross-lingual Retrieval for Iterative Self-Supervised Training
This work addresses the problem of enhancing cross-lingual capabilities for multilingual NLP applications, offering incremental improvements over existing methods.
The paper tackled improving cross-lingual alignment in multilingual models by introducing CRISS, an iterative self-supervised training method that mines and trains on sentence pairs, achieving state-of-the-art unsupervised machine translation results with an average 2.4 BLEU improvement on 9 language directions and a 21.5% accuracy boost on the Tatoeba retrieval task.
Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach -- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.