Domain-Adaptive Pretraining Methods for Dialogue Understanding
This work addresses the problem of enhancing dialogue systems for NLP practitioners, but it is incremental as it builds on existing pretraining methods.
The paper tackled the problem of improving dialogue understanding by evaluating domain-adaptive pretraining objectives, including a novel one for predicate-argument relations, and achieved new state-of-the-art performances on two challenging tasks.
Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.