Semantic-based Pre-training for Dialogue Understanding
This work addresses the challenge of improving dialogue understanding for AI systems by incorporating explicit semantic knowledge, representing an incremental advance in pre-training methods for dialogue tasks.
The paper tackled the problem of pre-trained language models being weak in understanding the main semantic meaning of dialogue contexts by proposing a semantic-based pre-training framework using Abstract Meaning Representation (AMR) to capture core semantic information, resulting in superior performance on both chit-chat and task-oriented dialogue understanding tasks.
Pre-trained language models have made great progress on dialogue tasks. However, these models are typically trained on surface dialogue text, thus are proven to be weak in understanding the main semantic meaning of a dialogue context. We investigate Abstract Meaning Representation (AMR) as explicit semantic knowledge for pre-training models to capture the core semantic information in dialogues during pre-training. In particular, we propose a semantic-based pre-training framework that extends the standard pre-training framework (Devlin et al., 2019) by three tasks for learning 1) core semantic units, 2) semantic relations and 3) the overall semantic representation according to AMR graphs. Experiments on the understanding of both chit-chats and task-oriented dialogues show the superiority of our model. To our knowledge, we are the first to leverage a deep semantic representation for dialogue pre-training.