EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues
This work addresses the need for consistent and automated empathy evaluation in conversational AI, reducing reliance on subjective human assessments.
The paper tackles the challenge of automatically measuring empathy in open-domain dialogues by proposing EMP-EVAL, a framework that achieves results comparable to human judgments, as demonstrated by its correlation with human preferences.
Measuring empathy in conversation can be challenging, as empathy is a complex and multifaceted psychological construct that involves both cognitive and emotional components. Human evaluations can be subjective, leading to inconsistent results. Therefore, there is a need for an automatic method for measuring empathy that reduces the need for human evaluations. In this paper, we proposed a novel approach EMP-EVAL, a simple yet effective automatic empathy evaluation method. The proposed technique takes the influence of Emotion, Cognitive and Emotional empathy. To the best knowledge, our work is the first to systematically measure empathy without the human-annotated provided scores. Experimental results demonstrate that our metrics can correlate with human preference, achieving comparable results with human judgments.