MINERS: Multilingual Language Models as Semantic Retrievers
This work addresses the need for better evaluation of multilingual models in semantic retrieval, particularly for low-resource languages, but it is incremental as it builds on existing embedding and retrieval techniques.
The paper tackles the problem of evaluating multilingual language models for semantic retrieval tasks, introducing the MINERS benchmark and showing that using embeddings for retrieval achieves competitive performance with state-of-the-art methods without fine-tuning.
Words have been represented in a high-dimensional vector space that encodes their semantic similarities, enabling downstream applications such as retrieving synonyms, antonyms, and relevant contexts. However, despite recent advances in multilingual language models (LMs), the effectiveness of these models' representations in semantic retrieval contexts has not been comprehensively explored. To fill this gap, this paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual LMs in semantic retrieval tasks, including bitext mining and classification via retrieval-augmented contexts. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches, without requiring any fine-tuning.