Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom
This work addresses the challenge of LLMs comprehending non-literal meaning in conversations, which is crucial for human-like social communication, though it is incremental as it focuses on a specific dataset and task.
The authors tackled the problem of whether large language models (LLMs) understand conversational implicature by creating SwordsmanImp, a Chinese multi-turn dialogue dataset from a sitcom, and testing eight LLMs on multiple-choice and explanation tasks. They found that GPT-4 achieved human-level accuracy (94%) on multiple-choice questions, while other models performed worse (20-60%), and most models scored low on explanation reasonability except GPT-4.
Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.