Understand the Implication: Learning to Think for Pragmatic Understanding
This addresses the challenge of enhancing LLMs' ability to infer implicit meaning in communication, which is crucial for social cognition, though it appears incremental in its approach.
The paper tackles the problem of improving LLMs' pragmatic understanding by introducing a novel dataset with explicit reasoning annotations and demonstrating that thought-based learning improves accuracy by 11.12% across model families, with transfer learning showing a 16.10% improvement on unseen pragmatic tasks.
Pragmatics, the ability to infer meaning beyond literal interpretation, is crucial for social cognition and communication. While LLMs have been benchmarked for their pragmatic understanding, improving their performance remains underexplored. Existing methods rely on annotated labels but overlook the reasoning process humans naturally use to interpret implicit meaning. To bridge this gap, we introduce a novel pragmatic dataset, ImpliedMeaningPreference, that includes explicit reasoning (thoughts) for both correct and incorrect interpretations. Through preference-tuning and supervised fine-tuning, we demonstrate that thought-based learning significantly enhances LLMs' pragmatic understanding, improving accuracy by 11.12% across model families. We further discuss a transfer-learning study where we evaluate the performance of thought-based training for the other tasks of pragmatics (presupposition, deixis) that are not seen during the training time and observe an improvement of 16.10% compared to label-trained models.