UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
This work identifies specific weaknesses in LLMs' linguistic reasoning for low-resource languages, which is incremental but useful for improving language modeling.
The study analyzed LLMs' performance on 629 Linguistics Olympiad puzzles across 41 low-resource languages, finding they struggle with higher morphological complexity and perform better on puzzles with English-like features, while splitting words into morphemes improved solvability.
Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages.