Benchmarking Machine Reading Comprehension: A Psychological Perspective
This is an incremental position paper that aims to improve MRC benchmarking for researchers by incorporating psychological principles to enhance explainability and validity.
The paper addresses the lack of explainability in machine reading comprehension (MRC) by proposing a theoretical framework based on psychology and psychometrics to design better MRC datasets, concluding that future datasets should evaluate coherent representation and include shortcut-proof questions with explanations.
Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design.