Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena
This work addresses a problem for linguists and NLP researchers by providing a scalable method to evaluate LLMs on rare constructions, though it is incremental in applying existing NLP tools to a new domain.
The authors tackled the challenge of evaluating Large Language Models' understanding of rare linguistic phenomena, specifically the caused-motion construction, by developing a hybrid human-LLM pipeline to construct a corpus at scale, and found that all tested models struggled with the motion component, with accuracy rates below 50% in some cases.
Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact that ``sneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.