Linguini: A benchmark for language-agnostic linguistic reasoning
This provides a new benchmark for assessing language-agnostic reasoning in AI, which is incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating language models' linguistic reasoning skills without language-specific knowledge by creating a benchmark from low-resource languages, finding that all models scored below 25% accuracy with a significant gap between proprietary (24.05%) and open models (8.84%).
We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages, extracted from the International Linguistic Olympiad corpus. To attain high accuracy on this benchmark, models don't need previous knowledge of the tested language, as all the information needed to solve the linguistic puzzle is presented in the context. We find that, while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models, with the best-performing proprietary model at 24.05% and the best-performing open model at 8.84%.