How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments
This work addresses the challenge of effectively assessing grammatical knowledge in LLMs for researchers and practitioners, offering incremental improvements over existing methods.
The study tackled the problem of accurately measuring grammatical knowledge in large language models (LLMs) by comparing nine judgment methods for acceptability judgments, finding that two methods—in-template LP and Yes/No probability computing—achieved higher accuracy than conventional approaches, with ensembling further improving performance.
The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches directly compare sentence probabilities assigned by LMs, but recent large language models (LLMs) are trained to perform tasks via prompting, and thus, the raw probabilities they assign may not fully reflect their grammatical knowledge. In this study, we attempt to derive more accurate acceptability judgments from LLMs using prompts and templates. Through extensive experiments in English and Chinese, we compare nine judgment methods and find two of them, a probability readout method -- in-template LP and a prompt-based method -- Yes/No probability computing, achieve higher accuracy than the conventional ones. Our analysis reveals that these methods excel in different linguistic phenomena, suggesting they access different aspects of LLMs' knowledge. We also find that ensembling the two methods outperforms single methods. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for assessing grammatical knowledge in LLMs.