Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
This addresses the problem of LLMs' limited social reasoning for applications in human-AI interactions, highlighting significant cross-lingual performance gaps and biases, though it is incremental as it builds on existing evaluation frameworks.
The paper tackled evaluating LLMs' social reasoning in interpersonal contexts by introducing the SCRIPTS dataset with 1k dialogues in English and Korean, finding that proprietary LLMs achieved 75-80% accuracy in English but dropped to 58-69% in Korean, with models selecting unlikely relationships in 10-25% of responses.
As large language models (LLMs) are increasingly used in human-AI interactions, their social reasoning capabilities in interpersonal contexts are critical. We introduce SCRIPTS, a 1k-dialogue dataset in English and Korean, sourced from movie scripts. The task involves evaluating models' social reasoning capability to infer the interpersonal relationships (e.g., friends, sisters, lovers) between speakers in each dialogue. Each dialogue is annotated with probabilistic relational labels (Highly Likely, Less Likely, Unlikely) by native (or equivalent) Korean and English speakers from Korea and the U.S. Evaluating nine models on our task, current proprietary LLMs achieve around 75-80% on the English dataset, whereas their performance on Korean drops to 58-69%. More strikingly, models select Unlikely relationships in 10-25% of their responses. Furthermore, we find that thinking models and chain-of-thought prompting, effective for general reasoning, provide minimal benefits for social reasoning and occasionally amplify social biases. Our findings reveal significant limitations in current LLMs' social reasoning capabilities, highlighting the need for efforts to develop socially-aware language models.