Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish
This work addresses the gap in grammar-focused evaluation protocols for low-resource languages, providing a systematic framework to assess if large language models genuinely comprehend grammatical structure, though it is incremental in applying existing methods to a new domain.
The study tackled the problem of evaluating grammatical understanding in large language models, especially for low-resource languages like Luxembourgish, by proposing a grammar-book-guided evaluation pipeline and found that larger models perform well overall but struggle with morphology and syntax, showing only a weak correlation between translation performance and grammatical competence.
Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a given language. In natural language processing, there remains a notable scarcity of grammar focused evaluation protocols, a gap that is even more pronounced for low-resource languages. Moreover, the extent to which large language models genuinely comprehend grammatical structure, especially the mapping between syntactic structures and meanings, remains under debate. To investigate this issue, we propose a Grammar Book Guided evaluation pipeline intended to provide a systematic and generalizable framework for grammar evaluation consisting of four key stages, and in this work we take Luxembourgish as a case study. The results show a weak positive correlation between translation performance and grammatical understanding, indicating that strong translations do not necessarily imply deep grammatical competence. Larger models perform well overall due to their semantic strength but remain weak in morphology and syntax, struggling particularly with Minimal Pair tasks, while strong reasoning ability offers a promising way to enhance their grammatical understanding.