DKAF: KB Arbitration for Learning Task-Oriented Dialog Systems with Dialog-KB Inconsistencies
This addresses a practical issue for developers of task-oriented dialog systems in dynamic environments, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of training task-oriented dialog agents when training dialogs contain facts conflicting with the latest knowledge base snapshot, which can confuse learning algorithms. They propose a Dialog-KB Arbitration Framework (DKAF) that predicts contemporary KB snapshots for each dialog to reduce inconsistencies, showing improved performance over baselines on two modified datasets.
Task-oriented dialog (TOD) agents often ground their responses on external knowledge bases (KBs). These KBs can be dynamic and may be updated frequently. Existing approaches for learning TOD agents assume the KB snapshot contemporary to each individual dialog is available during training. However, in real-world scenarios, only the latest KB snapshot is available during training and as a result, the train dialogs may contain facts conflicting with the latest KB. These dialog-KB inconsistencies in the training data may potentially confuse the TOD agent learning algorithm. In this work, we define the novel problem of learning a TOD agent with dialog-KB inconsistencies in the training data. We propose a Dialog-KB Arbitration Framework (DKAF) which reduces the dialog-KB inconsistencies by predicting the contemporary KB snapshot for each train dialog. These predicted KB snapshots are then used for training downstream TOD agents. As there are no existing datasets with dialog-KB inconsistencies, we systematically introduce inconsistencies in two publicly available dialog datasets. We show that TOD agents trained with DKAF perform better than existing baselines on both these datasets