A Review of Evaluation Techniques for Social Dialogue Systems
This addresses the challenge of evaluating social dialogue systems for researchers and developers, but it is incremental as it reviews existing methods without proposing new solutions.
The paper reviews evaluation methods for social dialogue systems, highlighting that current automatic metrics fail to account for context, multiple valid replies, and human perceptions.
In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.