MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
This provides a new benchmark for assessing LLMs' multilingual linguistic abilities, particularly for low-resource languages, though it is incremental as it builds on existing resources.
The authors tackled the problem of evaluating large language models (LLMs) on linguistic tasks by introducing MultiBLiMP 1.0, a benchmark covering 101 languages with over 128,000 minimal pairs, which revealed shortcomings in current models for low-resource languages.
We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.