FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows
This work addresses the challenge of developing better automatic metrics for open-domain dialogue systems, which is crucial for researchers and developers in conversational AI, though it is incremental in extending existing dialog act concepts.
The paper tackles the problem of automatic dialogue evaluation by introducing segment acts, a finer-grained extension of dialog acts, and proposes FlowEval, a consensus-based framework that uses segment act flows for reference-free evaluation. Experiments on three benchmark datasets show its effectiveness, outperforming strong baselines.
Despite recent progress in open-domain dialogue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly explicitly modeled in previous methods. However, defined at the utterance level in general, dialog act is of coarse granularity, as an utterance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Extensive experiments against strong baselines on three benchmark datasets demonstrate the effectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation.