Building Russian Benchmark for Evaluation of Information Retrieval Models
This work addresses the problem of standardized evaluation for Russian-language information retrieval, which is incremental as it adapts existing benchmark concepts to a new language.
The authors tackled the lack of a comprehensive benchmark for evaluating information retrieval models in Russian by introducing RusBEIR, a benchmark with 17 datasets, and found that neural models like mE5-large and BGE-M3 outperform lexical models in most cases, though they struggle with long-document retrieval.
We introduce RusBEIR, a comprehensive benchmark designed for zero-shot evaluation of information retrieval (IR) models in the Russian language. Comprising 17 datasets from various domains, it integrates adapted, translated, and newly created datasets, enabling systematic comparison of lexical and neural models. Our study highlights the importance of preprocessing for lexical models in morphologically rich languages and confirms BM25 as a strong baseline for full-document retrieval. Neural models, such as mE5-large and BGE-M3, demonstrate superior performance on most datasets, but face challenges with long-document retrieval due to input size constraints. RusBEIR offers a unified, open-source framework that promotes research in Russian-language information retrieval.