Non-literal Understanding of Number Words by Language Models
This addresses the issue of AI-human differences in pragmatic reasoning for improving language understanding capabilities, though it is incremental as it builds on existing frameworks.
The paper tackled the problem of whether large language models (LLMs) interpret numbers non-literally like humans, and found that LLMs diverge from human interpretation, but chain-of-thought prompting inspired by an RSA model made their interpretations more human-like.
Humans naturally interpret numbers non-literally, effortlessly combining context, world knowledge, and speaker intent. We investigate whether large language models (LLMs) interpret numbers similarly, focusing on hyperbole and pragmatic halo effects. Through systematic comparison with human data and computational models of pragmatic reasoning, we find that LLMs diverge from human interpretation in striking ways. By decomposing pragmatic reasoning into testable components, grounded in the Rational Speech Act framework, we pinpoint where LLM processing diverges from human cognition -- not in prior knowledge, but in reasoning with it. This insight leads us to develop a targeted solution -- chain-of-thought prompting inspired by an RSA model makes LLMs' interpretations more human-like. Our work demonstrates how computational cognitive models can both diagnose AI-human differences and guide development of more human-like language understanding capabilities.