Schema-Guided Dialogue State Tracking Task at DSTC8
This task addresses the problem of scalable and generalizable dialogue state tracking for virtual assistants, though it is incremental as it builds on existing challenges and datasets.
The paper introduced the Schema-Guided Dialogue State Tracking task at DSTC8, aiming to develop models for large-scale virtual assistants with data-efficient joint modeling and zero-shot generalization, using a dataset of over 16,000 dialogues across 16 domains and a baseline model, where 25 participating teams significantly outperformed the baseline with neural network approaches.
This paper gives an overview of the Schema-Guided Dialogue State Tracking task of the 8th Dialogue System Technology Challenge. The goal of this task is to develop dialogue state tracking models suitable for large-scale virtual assistants, with a focus on data-efficient joint modeling across domains and zero-shot generalization to new APIs. This task provided a new dataset consisting of over 16000 dialogues in the training set spanning 16 domains to highlight these challenges, and a baseline model capable of zero-shot generalization to new APIs. Twenty-five teams participated, developing a range of neural network models, exceeding the performance of the baseline model by a very high margin. The submissions incorporated a variety of pre-trained encoders and data augmentation techniques. This paper describes the task definition, dataset and evaluation methodology. We also summarize the approach and results of the submitted systems to highlight the overall trends in the state-of-the-art.