To Test Machine Comprehension, Start by Defining Comprehension
This work addresses the foundational problem of defining comprehension in AI for researchers, but it is incremental as it focuses on narrative texts without introducing new methods.
The paper argues that existing machine reading comprehension tasks lack a clear definition of comprehension and proposes a 'Template of Understanding' for short narratives, with experiments showing current systems fail to meet this standard.
Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension -- a "Template of Understanding" -- for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.