CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation
This provides a comprehensive evaluation framework for Chinese goal-oriented dialog systems, addressing gaps between academic benchmarks and real-world spoken scenarios.
The authors tackled the problem of evaluating goal-oriented dialog systems by creating CGoDial, a large-scale Chinese benchmark containing 96,763 dialog sessions and 574,949 dialog turns across three datasets with different knowledge sources, which enables assessment of model capabilities in general prediction, fast adaptability, and robustness.
Practical dialog systems need to deal with various knowledge sources, noisy user expressions, and the shortage of annotated data. To better solve the above problems, we propose CGoDial, new challenging and comprehensive Chinese benchmark for multi-domain Goal-oriented Dialog evaluation. It contains 96,763 dialog sessions and 574,949 dialog turns totally, covering three datasets with different knowledge sources: 1) a slot-based dialog (SBD) dataset with table-formed knowledge, 2) a flow-based dialog (FBD) dataset with tree-formed knowledge, and a retrieval-based dialog (RBD) dataset with candidate-formed knowledge. To bridge the gap between academic benchmarks and spoken dialog scenarios, we either collect data from real conversations or add spoken features to existing datasets via crowd-sourcing. The proposed experimental settings include the combinations of training with either the entire training set or a few-shot training set, and testing with either the standard test set or a hard test subset, which can assess model capabilities in terms of general prediction, fast adaptability and reliable robustness.