Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking
This work addresses a methodological gap in evaluating dialogue state tracking systems, which is important for researchers and practitioners in natural language processing, though it is incremental as it focuses on improving evaluation rather than introducing new models.
The authors identified a critical limitation in existing evaluation metrics for dialogue state tracking, particularly in the MultiWOZ dataset, where metrics fail to properly assess belief states as they accumulate across dialogue turns. They proposed relative slot accuracy as a complementary metric that is independent of predefined slots and assigns scores based on dialogue turn, encouraging the use of diverse metrics for more realistic evaluations.
Dialogue state tracking (DST) aims to extract essential information from multi-turn dialogue situations and take appropriate actions. A belief state, one of the core pieces of information, refers to the subject and its specific content, and appears in the form of domain-slot-value. The trained model predicts "accumulated" belief states in every turn, and joint goal accuracy and slot accuracy are mainly used to evaluate the prediction; however, we specify that the current evaluation metrics have a critical limitation when evaluating belief states accumulated as the dialogue proceeds, especially in the most used MultiWOZ dataset. Additionally, we propose relative slot accuracy to complement existing metrics. Relative slot accuracy does not depend on the number of predefined slots, and allows intuitive evaluation by assigning relative scores according to the turn of each dialogue. This study also encourages not solely the reporting of joint goal accuracy, but also various complementary metrics in DST tasks for the sake of a realistic evaluation.