Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval
This work addresses the generalization of multilingual encoders to unsupervised cross-lingual information retrieval tasks, providing insights for researchers and practitioners in multilingual NLP, though it is incremental as it builds on existing encoder paradigms.
The study evaluated multilingual text encoders like mBERT and XLM for unsupervised cross-lingual retrieval, finding that they do not significantly outperform cross-lingual word embeddings for document-level retrieval but can achieve state-of-the-art performance for sentence-level retrieval with task-specialized variants.
Pretrained multilingual text encoders based on neural Transformer architectures, such as multilingual BERT (mBERT) and XLM, have achieved strong performance on a myriad of language understanding tasks. Consequently, they have been adopted as a go-to paradigm for multilingual and cross-lingual representation learning and transfer, rendering cross-lingual word embeddings (CLWEs) effectively obsolete. However, questions remain to which extent this finding generalizes 1) to unsupervised settings and 2) for ad-hoc cross-lingual IR (CLIR) tasks. Therefore, in this work we present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks across a large number of language pairs. In contrast to supervised language understanding, our results indicate that for unsupervised document-level CLIR -- a setup with no relevance judgments for IR-specific fine-tuning -- pretrained encoders fail to significantly outperform models based on CLWEs. For sentence-level CLIR, we demonstrate that state-of-the-art performance can be achieved. However, the peak performance is not met using the general-purpose multilingual text encoders `off-the-shelf', but rather relying on their variants that have been further specialized for sentence understanding tasks.