Assessing Dialogue Systems with Distribution Distances
This addresses the need for more accurate automatic evaluation metrics for dialogue systems, though it is incremental as it builds on existing distribution distance methods.
The paper tackled the problem of evaluating dialogue systems by proposing distribution-wise metrics (FBD and PRD) that measure the distance between generated and real-world conversation distributions, resulting in better correlation with human judgments than existing metrics.
An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing automatic evaluation metrics are based on turn-level quality evaluation and use average scores for system-level comparison. In this paper, we propose to measure the performance of a dialogue system by computing the distribution-wise distance between its generated conversations and real-world conversations. Specifically, two distribution-wise metrics, FBD and PRD, are developed and evaluated. Experiments on several dialogue corpora show that our proposed metrics correlate better with human judgments than existing metrics.