RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
This provides a domain-specific tool for researchers and practitioners working on Russian language models to assess grammatical knowledge, though it is incremental as it extends minimal pair benchmarks to a new language.
The paper introduces RuBLiMP, a benchmark with 45k Russian sentence pairs for evaluating language models on grammatical phenomena, finding that models perform well on morphology and agreement but lag behind humans on structural relations, negation, transitivity, and tense.
Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models. However, existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena. This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and carefully curating test data. We describe the data collection protocol and present the results of evaluating 25 language models in various scenarios. We find that the widely used language models for Russian are sensitive to morphological and agreement-oriented contrasts but fall behind humans on phenomena requiring understanding of structural relations, negation, transitivity, and tense. RuBLiMP, the codebase, and other materials are publicly available.