An Evaluation Protocol for Generative Conversational Systems
This provides a standardized evaluation framework for researchers in conversational AI, though it is incremental as it builds on existing methods.
The paper tackles the lack of systematic evaluation for generative conversational models by proposing a protocol using head-to-head pairwise comparisons, finding that DialoGPT and Blender are superior systems based on analysis of ten models across five datasets.
There is a multitude of novel generative models for open-domain conversational systems; however, there is no systematic evaluation of different systems. Systematic comparisons require consistency in experimental design, evaluation sets, conversational systems and their outputs, and statistical analysis. We lay out a protocol for the evaluation of conversational models using head-to-head pairwise comparison. We analyze ten recent models that claim state-of-the-art performance using a paired head-to-head performance (win-loss-tie) on five evaluation datasets. Our findings show that DialoGPT and Blender are superior systems using Bradley-Terry model and TrueSkill ranking methods. These findings demonstrate the feasibility of our protocol to evaluate conversational agents and evaluation sets. Finally, we make all code and evaluations publicly available for researchers to compare their model to other state-of-the-art dialog models.