ExpMRC: Explainability Evaluation for Machine Reading Comprehension
This addresses the need for reliable MRC systems in real-life applications by providing a standardized evaluation for explainability, though it is incremental as it builds on existing datasets with new annotations.
The authors introduced ExpMRC, a benchmark for evaluating explainability in machine reading comprehension (MRC) systems, requiring both correct answers and evidence-based explanations across four datasets. Experimental results show that state-of-the-art models perform far below human levels, indicating the benchmark's challenge.
Achieving human-level performance on some of Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE$^+$, and C$^3$ with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art pre-trained language models to build baseline systems and adopt various unsupervised approaches to extract evidence without a human-annotated training set. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources will be available through https://github.com/ymcui/expmrc