Multi-dimensional Evaluation of Empathetic Dialog Responses
This work addresses the challenge of evaluating empathy in conversational AI for applications like customer service, though it is incremental as it builds on existing methods by adding a listener perspective.
The authors tackled the problem of evaluating empathy in dialogues by proposing a multi-dimensional framework that measures both expressed intents and perceived empathy, finding that perceived empathy correlates highly with dialogue satisfaction. They also developed instruction-finetuned classifiers based on Flan-T5 models, which outperformed prior methods and baselines in automated measurement.
Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents -- that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both \emph{expressed intents from the speaker's perspective} and \emph{perceived empathy from the listener's perspective}. We apply our analytical framework to examine internal customer-service dialogues. We find the two dimensions (expressed intent types and perceived empathy) are inter-connected, while perceived empathy has high correlations with dialogue satisfaction levels. To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on Flan-T5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method's strong performance.