Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering
This work addresses the challenge of processing complex dialogue structures for QA, offering incremental gains in a domain-specific setting.
The paper tackles the problem of understanding hierarchical contexts in multiparty dialogue for span-based question answering by introducing a novel transformer approach that uses pre-training tasks and multi-task learning, resulting in improvements of 3.8% and 1.4% over BERT and RoBERTa on the FriendsQA dataset.
We introduce a novel approach to transformers that learns hierarchical representations in multiparty dialogue. First, three language modeling tasks are used to pre-train the transformers, token- and utterance-level language modeling and utterance order prediction, that learn both token and utterance embeddings for better understanding in dialogue contexts. Then, multi-task learning between the utterance prediction and the token span prediction is applied to fine-tune for span-based question answering (QA). Our approach is evaluated on the FriendsQA dataset and shows improvements of 3.8% and 1.4% over the two state-of-the-art transformer models, BERT and RoBERTa, respectively.