How to Evaluate Your Dialogue Models: A Review of Approaches
This is an incremental survey that organizes existing knowledge for researchers and practitioners in dialogue systems.
The paper tackles the understudied problem of evaluating dialogue systems by providing a comprehensive survey that classifies evaluation methods into three categories and discusses benchmarks and open issues.
Evaluating the quality of a dialogue system is an understudied problem. The recent evolution of evaluation method motivated this survey, in which an explicit and comprehensive analysis of the existing methods is sought. We are first to divide the evaluation methods into three classes, i.e., automatic evaluation, human-involved evaluation and user simulator based evaluation. Then, each class is covered with main features and the related evaluation metrics. The existence of benchmarks, suitable for the evaluation of dialogue techniques are also discussed in detail. Finally, some open issues are pointed out to bring the evaluation method into a new frontier.