How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation
This work addresses the need for better evaluation benchmarks in dialogue systems, providing a large-scale dataset to improve metric robustness, though it is incremental as it builds on existing datasets.
The authors tackled the problem of evaluating automatic dialogue metrics by creating MMSMR, a massively multi-system multi-reference dataset, and found that current metrics are not robust proxies for human judgments, with correlations often below 0.5.
We release MMSMR, a Massively Multi-System MultiReference dataset to enable future work on metrics and evaluation for dialog. Automatic metrics for dialogue evaluation should be robust proxies for human judgments; however, the verification of robustness is currently far from satisfactory. To quantify the robustness correlation and understand what is necessary in a test set, we create and release an 8-reference dialog dataset by extending single-reference evaluation sets and introduce this new language learning conversation dataset. We then train 1750 systems and evaluate them on our novel test set and the DailyDialog dataset. We release the novel test set, and model hyper parameters, inference outputs, and metric scores for each system on a variety of datasets.