OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System
This work addresses a gap in task-oriented dialogue systems by integrating comprehension tasks into pre-training, potentially improving performance for developers and users, though it appears incremental as it builds on existing PCM frameworks.
The paper tackled the problem of pre-trained conversation models (PCMs) often neglecting dialogue comprehension tasks like question answering and summarization, which could enhance downstream performance; by proposing OmniDialog, a model pre-trained on 7 tasks from 15 datasets with over 3.2 million utterances, it showed efficacy in domain transfer, low-resource, and full-dataset scenarios, particularly excelling on hard samples such as long dialogues and lengthy responses.
Pre-trained conversation models (PCMs) have demonstrated remarkable results in task-oriented dialogue (TOD) systems. Many PCMs focus predominantly on dialogue management tasks like dialogue state tracking, dialogue generation tasks like response generation, or both. However, the existing PCMs seldom consider dialogue comprehension tasks, such as dialogue question answering and summarization tasks. These tasks allow PCMs to glean dialogue context from various angles. This observation naturally raises the question: Can the performance of downstream dialogue tasks be enhanced if a PCM is pre-trained on dialogue management, generation, and comprehension tasks? To investigate this, we proposed an Omnipotent Dialogue pre-training model (OmniDialog). It unifies these three dialogue tasks into a monolithic framework by multi-task learning, fostering inter-task communication. The pre-training corpus of OmniDialog spans $\mathbf{7}$ dialogue-focused tasks, drawing from $\mathbf{15}$ datasets and encompassing over $\mathbf{3.2}$ million dialogue utterances. To our knowledge, OmniDialog is a pioneering PCM pre-trained across dialogue management, generation, and comprehension domains. We evaluated its performance across four tasks: dialogue summarization, end-to-end dialogue modeling, dialogue state tracking, and intent classification. The results underscore its efficacy in domain transfer learning, low-resource, and full-dataset scenarios. Furthermore, to glean a nuanced understanding of OmniDialog's strengths and potential pitfalls, we designed a fine-grained analysis framework for dialogue-centric tasks. Experimental results show that the OmniDialog is good at hard samples, such as long dialogues and lengthy responses.