What Can String Probability Tell Us About Grammaticality?
This work addresses a fundamental debate in linguistics and AI about the interpretability of language models, with implications for evaluating their grammatical knowledge, though it is incremental in building on existing theoretical frameworks.
The paper tackles the problem of understanding what language models have learned about grammar by analyzing the relationship between string probability and grammaticality. It validates three predictions using 280K sentence pairs in English and Chinese, showing correlations in minimal pairs and poor separation between grammatical and ungrammatical strings.
What have language models (LMs) learned about grammar? This question remains hotly debated, with major ramifications for linguistic theory. However, since probability and grammaticality are distinct notions in linguistics, it is not obvious what string probabilities can reveal about an LM's underlying grammatical knowledge. We present a theoretical analysis of the relationship between grammar, meaning, and string probability, based on simple assumptions about the generative process of corpus data. Our framework makes three predictions, which we validate empirically using 280K sentence pairs in English and Chinese: (1) correlation between the probability of strings within minimal pairs, i.e., string pairs with minimal semantic differences; (2) correlation between models' and humans' deltas within minimal pairs; and (3) poor separation in probability space between unpaired grammatical and ungrammatical strings. Our analyses give theoretical grounding for using probability to learn about LMs' structural knowledge, and suggest directions for future work in LM grammatical evaluation.