Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking
This addresses evaluation inaccuracies for DST systems, especially in low-resource settings, but is incremental as it builds on existing metric frameworks.
The authors tackled the problem of biased evaluation metrics in Dialogue State Tracking (DST) by introducing Granular Change Accuracy (GCA), which reduces biases from uniform slot distribution and error positioning, showing particular promise for few-shot or zero-shot trained models.
Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations. They: i) erroneously presume a uniform distribution of slots throughout the dialog, ii) neglect to assign partial scores for individual turns, iii) frequently overestimate or underestimate performance by repeatedly counting the models' successful or failed predictions. To address these shortcomings, we introduce a novel metric: Granular Change Accuracy (GCA). GCA focuses on evaluating the predicted changes in dialogue state over the entire dialogue history. Benchmarking reveals that GCA effectively reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation. Notably, we find that these biases are particularly pronounced when evaluating few-shot or zero-shot trained models, becoming even more evident as the model's error rate increases. Hence, GCA offers significant promise, particularly for assessing models trained with limited resources. Our GCA implementation is a useful addition to the pool of DST metrics.