Exploring the Impact of Human Evaluator Group on Chat-Oriented Dialogue Evaluation
This work addresses the problem of inconsistent human evaluation practices in dialogue system research, which is incremental as it clarifies existing methodologies rather than introducing new ones.
The paper investigated how different human evaluator groups affect the assessment of chat-oriented dialogue systems, finding that Likert evaluations are robust to group changes while Pairwise evaluations are not, with minor differences observed.
Human evaluation has been widely accepted as the standard for evaluating chat-oriented dialogue systems. However, there is a significant variation in previous work regarding who gets recruited as evaluators. Evaluator groups such as domain experts, university students, and professional annotators have been used to assess and compare dialogue systems, although it is unclear to what extent the choice of an evaluator group can affect results. This paper analyzes the evaluator group impact on dialogue system evaluation by testing 4 state-of-the-art dialogue systems using 4 distinct evaluator groups. Our analysis reveals a robustness towards evaluator groups for Likert evaluations that is not seen for Pairwise, with only minor differences observed when changing evaluator groups. Furthermore, two notable limitations to this robustness are observed, which reveal discrepancies between evaluators with different levels of chatbot expertise and indicate that evaluator objectivity is beneficial for certain dialogue metrics.