DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension
This dataset addresses the problem of evaluating dialogue understanding for reading comprehension systems, particularly for English learners, and is incremental as it builds on existing reading comprehension datasets by focusing on multi-turn dialogues.
The authors introduced DREAM, a dialogue-based multiple-choice reading comprehension dataset with 10,197 questions from 6,444 dialogues, designed to challenge systems with non-extractive answers and reasoning beyond single sentences, and found that incorporating dialogue structure and world knowledge improved model performance.
We present DREAM, the first dialogue-based multiple-choice reading comprehension dataset. Collected from English-as-a-foreign-language examinations designed by human experts to evaluate the comprehension level of Chinese learners of English, our dataset contains 10,197 multiple-choice questions for 6,444 dialogues. In contrast to existing reading comprehension datasets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge. We apply several popular neural reading comprehension models that primarily exploit surface information within the text and find them to, at best, just barely outperform a rule-based approach. We next investigate the effects of incorporating dialogue structure and different kinds of general world knowledge into both rule-based and (neural and non-neural) machine learning-based reading comprehension models. Experimental results on the DREAM dataset show the effectiveness of dialogue structure and general world knowledge. DREAM will be available at https://dataset.org/dream/.