LingGym: How Far Are LLMs from Thinking Like Field Linguists?
This addresses the problem of assessing LLMs' linguistic generalization for low-resource language documentation, though it appears incremental as a new benchmark.
The paper introduces LingGym, a benchmark evaluating LLMs' meta-linguistic reasoning using data from 18 diverse languages, showing that structured linguistic cues improve performance in word-gloss inference tasks.
This paper introduces LingGym, a new benchmark that evaluates LLMs' capacity for meta-linguistic reasoning using Interlinear Glossed Text (IGT) and grammatical descriptions extracted from 18 typologically diverse reference grammars. Unlike previous work that focuses on specific downstream tasks, we assess whether LLMs can generalize linguistic inference across low-resource languages and structures not seen during training. We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context using varying levels of linguistic information (e.g., glosses, grammatical explanations, translations). Our results show that incorporating structured linguistic cues leads to consistent improvements in reasoning performance across all models. This work highlights both the promise and current limitations of using LLMs for typologically informed linguistic analysis and low-resource language documentation.