EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World
This addresses the need for better evaluation in emotional support AI, shifting focus from generic empathy to user-aware systems, though it is incremental as it builds on existing paradigms with a new benchmark.
The paper tackled the problem of evaluating emotional support conversations by introducing EmoHarbor, an automated framework that simulates users' internal worlds to assess personalized support, revealing that advanced LLMs fail to tailor responses to individual contexts despite excelling at generic empathy.
Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users' unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-Agent architecture that decomposes users' internal processes into three specialized roles, enabling agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.