CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems
This addresses the challenge of accurate automated evaluation for dialogue systems, which is crucial for developers and researchers, though it appears incremental as it builds on existing metric approaches.
The authors tackled the problem of automatically evaluating response relevance in open-domain dialogue systems, where existing metrics often misalign with human judgments, by proposing CausalScore, a metric that measures causal strength between dialogue history and responses, and it significantly outperformed state-of-the-art metrics in aligning with human judgments.
Automatically evaluating the quality of responses in open-domain dialogue systems is a challenging but crucial task. Current evaluation metrics often fail to align with human judgments, especially when assessing responses that are grammatically correct. To address this issue, we propose a novel metric, called CausalScore, which assesses the relevance of responses by measuring the causal strength between dialogue histories and responses. The causal strength is estimated by utilizing both unconditional dependence and conditional dependencies from the dialogue history to responses. We compare our metric with the existing competitive metrics in terms of their alignment with human judgements. Our experimental results demonstrate that CausalScore significantly surpasses existing state-of-the-art metrics by aligning better with human judgements. Additionally, we collect a new dialogue dataset CGDIALOG+ with human-annotated causal relations and a set of pairwise human judgements to facilitate the development of future automatic metrics.