Pragmatic inference of scalar implicature by LLMs
This addresses the problem of understanding pragmatic language capabilities in LLMs for researchers in computational linguistics and AI, but it is incremental as it applies existing methods to analyze known models.
This study investigated how BERT and GPT-2 perform pragmatic inference of scalar implicature like 'some', finding that both models interpret 'some' as 'not all' without context, aligning with human processing, but GPT-2 struggled with contextual cues while BERT remained consistent.
This study investigates how Large Language Models (LLMs), particularly BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), engage in pragmatic inference of scalar implicature, such as some. Two sets of experiments were conducted using cosine similarity and next sentence/token prediction as experimental methods. The results in experiment 1 showed that, both models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. In experiment 2, in which Question Under Discussion (QUD) was presented as a contextual cue, BERT showed consistent performance regardless of types of QUDs, while GPT-2 encountered processing difficulties since a certain type of QUD required pragmatic inference for implicature. The findings revealed that, in terms of theoretical approaches, BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model (Levinson, 2000). In contrast, GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model (Sperber and Wilson, 2002).