Evaluating Large Language Models for Cross-Lingual Retrieval
This work addresses the lack of systematic evaluation for LLMs in cross-lingual retrieval, which is important for improving search systems in multilingual contexts, though it is incremental as it builds on existing multi-stage IR paradigms.
The paper tackled the problem of evaluating large language models (LLMs) for cross-lingual information retrieval (CLIR) by systematically comparing them in a two-stage setup, revealing that without machine translation, current state-of-the-art rerankers perform poorly, and showing that multilingual bi-encoders as first-stage retrievers can achieve further gains.
Multi-stage information retrieval (IR) has become a widely-adopted paradigm in search. While Large Language Models (LLMs) have been extensively evaluated as second-stage reranking models for monolingual IR, a systematic large-scale comparison is still lacking for cross-lingual IR (CLIR). Moreover, while prior work shows that LLM-based rerankers improve CLIR performance, their evaluation setup relies on lexical retrieval with machine translation (MT) for the first stage. This is not only prohibitively expensive but also prone to error propagation across stages. Our evaluation on passage-level and document-level CLIR reveals that further gains can be achieved with multilingual bi-encoders as first-stage retrievers and that the benefits of translation diminishes with stronger reranking models. We further show that pairwise rerankers based on instruction-tuned LLMs perform competitively with listwise rerankers. To the best of our knowledge, we are the first to study the interaction between retrievers and rerankers in two-stage CLIR with LLMs. Our findings reveal that, without MT, current state-of-the-art rerankers fall severely short when directly applied in CLIR.