Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation
For NLP researchers, this work highlights a gap in LLMs' pragmatic understanding of belief updates, though it is incremental as it applies existing evaluation methods to a new dataset.
The paper evaluates LLMs' ability to recognize unspoken beliefs via implicatures and understand their updates through cancellation, finding that LLMs lag behind humans, especially in natural scenarios, with successes partly due to reliance on prior beliefs.
Human language is driven by unspoken beliefs and belief updates, making these critical to model for successful communication between large language models (LLMs) and their users. In this paper, we evaluate the ability of LLMs to recognize unspoken beliefs made through implicatures and to understand their updates through implicature cancellation: the pragmatic phenomenon whereby an utterance's implied meaning is weakened or negated. We create the first expert-annotated implicature cancellation dataset, [DatasetName], crowdsourced for human judgements of implicatures and their corresponding cancellations. We find that LLM belief update understanding lags behind that of humans, especially in more naturally-occurring scenarios. Additional control experiments suggest that successes in LLM belief updates may stem in part from a reliance on prior beliefs, and that failures in belief updates may depend on their type and on their form. Overall, our study suggests that current LLMs have not yet reached human-level understanding of unspoken beliefs and belief updates. Code and data are available at https://github.com/cesare-spinoso/ImplicatureX.