Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining
This addresses the challenge of multilingual alignment for researchers in NLP, offering a transferable method that reduces the need for extensive multilingual data, though it is incremental as it builds on existing techniques like XLM-R.
The paper tackles the problem of learning cross-lingual sentence representations for unsupervised parallel text mining, showing that models trained on one language pair can transfer to other pairs, with the unsupervised model achieving state-of-the-art retrieval performance and the supervised model approaching multilingual supervised performance.
This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilingually more aligned representations. We thus introduce dual-pivot transfer: training on one language pair and evaluating on other pairs. To study this theory, we design unsupervised models trained on unpaired sentences and single-pair supervised models trained on bitexts, both based on the unsupervised language model XLM-R with its parameters frozen. The experiments evaluate the models as universal sentence encoders on the task of unsupervised bitext mining on two datasets, where the unsupervised model reaches the state of the art of unsupervised retrieval, and the alternative single-pair supervised model approaches the performance of multilingually supervised models. The results suggest that bilingual training techniques as proposed can be applied to get sentence representations with multilingual alignment.