MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems
This work addresses the problem of systematic evaluation for emotional companionship dialogue systems, which is crucial for developers aiming to enhance user experience, though it is incremental as it builds on existing dialogue system frameworks.
The paper tackles the lack of clear definitions and evaluation standards for Emotional Companionship Dialogue Systems (ECDs) by proposing a formal definition and designing MoodBench 1.0, the first ECD evaluation benchmark, which demonstrated excellent discriminant validity in evaluating 30 mainstream models and revealed their shortcomings in deep emotional companionship.
With the rapid development of Large Language Models, dialogue systems are shifting from information tools to emotional companions, heralding the era of Emotional Companionship Dialogue Systems (ECDs) that provide personalized emotional support for users. However, the field lacks clear definitions and systematic evaluation standards for ECDs. To address this, we first propose a definition of ECDs with formal descriptions. Then, based on this theory and the design principle of "Ability Layer-Task Layer (three level)-Data Layer-Method Layer", we design and implement the first ECD evaluation benchmark - MoodBench 1.0. Through extensive evaluations of 30 mainstream models, we demonstrate that MoodBench 1.0 has excellent discriminant validity and can effectively quantify the differences in emotional companionship abilities among models. Furthermore, the results reveal current models' shortcomings in deep emotional companionship, guiding future technological optimization and significantly aiding developers in enhancing ECDs' user experience.