Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions
This work addresses a controversial issue in experimental pragmatics for linguists and AI researchers, but it is incremental as it applies existing methods to new data without introducing novel techniques.
The paper tackled the problem of comparing inferences from plain and embedded disjunctions with regular scalar implicatures by testing predictions from state-of-the-art large language models using human experimental paradigms, finding that the best models mostly align with human results in both large differences and fine-grained distinctions.
Human communication is based on a variety of inferences that we draw from sentences, often going beyond what is literally said. While there is wide agreement on the basic distinction between entailment, implicature, and presupposition, the status of many inferences remains controversial. In this paper, we focus on three inferences of plain and embedded disjunctions, and compare them with regular scalar implicatures. We investigate this comparison from the novel perspective of the predictions of state-of-the-art large language models, using the same experimental paradigms as recent studies investigating the same inferences with humans. The results of our best performing models mostly align with those of humans, both in the large differences we find between those inferences and implicatures, as well as in fine-grained distinctions among different aspects of those inferences.