SocialNLI: A Dialogue-Centric Social Inference Dataset
This work addresses the need for better social inference capabilities in AI assistants, which is crucial for adept human-AI interaction, but it is incremental as it focuses on dataset creation and evaluation rather than a novel method.
The authors tackled the problem of large language and reasoning models struggling with sophisticated social phenomena like sarcasm and irony in dialogue by introducing SocialNLI, the first social dialogue inference dataset, which includes dialogue transcripts with complex social nuances, inferences, likelihood scores, and explanations, and they evaluated model theory-of-mind ability through multi-step counterfactual reasoning.
Making theory-of-mind inferences from human dialogue is a strong indicator of a model's underlying social abilities, which are fundamental for adept AI assistants. However, large language and reasoning models struggle to understand sophisticated social phenomena in transcript data, such as sarcasm and irony. To assess the weaknesses of current models and to identify their solutions, we introduce SocialNLI (SoNLI) -- the first social dialogue inference dataset. SoNLI consists of a collection of dialogue transcripts hand-picked to center complex social nuances like irony and sarcasm, paired with inferences, corresponding likelihood scores, and human-written explanations. We explore social inference analysis as a facet of theory-of-mind, and evaluate LLM and reasoning model theory-of-mind ability through multi-step counterfactual reasoning.