IIRC: A Dataset of Incomplete Information Reading Comprehension Questions
This addresses the need for evaluating systems on identifying information gaps and locating sources in multi-document reading comprehension, though it is incremental as it builds on existing datasets and modeling work.
The authors tackled the problem of reading comprehension with incomplete information by introducing the IIRC dataset, which contains over 13K questions based on Wikipedia paragraphs that require accessing linked documents for answers, and they found that a baseline model achieved 31.1% F1 compared to an estimated human performance of 88.4%.
Humans often have to read multiple documents to address their information needs. However, most existing reading comprehension (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system's performance at identifying a potential lack of sufficient information and locating sources for that information. To fill this gap, we present a dataset, IIRC, with more than 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. The questions were written by crowd workers who did not have access to any of the linked documents, leading to questions that have little lexical overlap with the contexts where the answers appear. This process also gave many questions without answers, and those that require discrete reasoning, increasing the difficulty of the task. We follow recent modeling work on various reading comprehension datasets to construct a baseline model for this dataset, finding that it achieves 31.1% F1 on this task, while estimated human performance is 88.4%. The dataset, code for the baseline system, and a leaderboard can be found at https://allennlp.org/iirc.