Evaluating statistical language models as pragmatic reasoners
This work addresses the challenge of using statistical language models for pragmatic inference, which is important for improving semantic parsing applications, though it is incremental in extending existing LLM evaluations to pragmatic contexts.
The paper tackled the problem of evaluating large language models (LLMs) as pragmatic reasoners by testing their ability to infer meanings from pragmatic utterances, such as those involving gradable adjectives and composition with negation, and found that LLMs can produce human-like distributions for complex utterances but struggle with negation.
The relationship between communicated language and intended meaning is often probabilistic and sensitive to context. Numerous strategies attempt to estimate such a mapping, often leveraging recursive Bayesian models of communication. In parallel, large language models (LLMs) have been increasingly applied to semantic parsing applications, tasked with inferring logical representations from natural language. While existing LLM explorations have been largely restricted to literal language use, in this work, we evaluate the capacity of LLMs to infer the meanings of pragmatic utterances. Specifically, we explore the case of threshold estimation on the gradable adjective ``strong'', contextually conditioned on a strength prior, then extended to composition with qualification, negation, polarity inversion, and class comparison. We find that LLMs can derive context-grounded, human-like distributions over the interpretations of several complex pragmatic utterances, yet struggle composing with negation. These results inform the inferential capacity of statistical language models, and their use in pragmatic and semantic parsing applications. All corresponding code is made publicly available (https://github.com/benlipkin/probsem/tree/CogSci2023).