Connecting Attributions and QA Model Behavior on Realistic Counterfactuals
This work addresses the challenge of interpreting model behavior for users in reading comprehension, though it is incremental as it builds on existing attribution methods.
The paper tackled the problem of evaluating how well attribution techniques align with counterfactual assumptions in reading comprehension models, finding that pairwise attributions outperform token-level attributions across different settings, with a proposed modification achieving the best performance.
When a model attribution technique highlights a particular part of the input, a user might understand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model's prediction might change as well. This paper investigates how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). RC is a particularly challenging test case, as token-level attributions that have been extensively studied in other NLP tasks such as sentiment analysis are less suitable to represent the reasoning that RC models perform. We construct counterfactual sets for three different RC settings, and through heuristics that can connect attribution methods' outputs to high-level model behavior, we can evaluate how useful different attribution methods and even different formats are for understanding counterfactuals. We find that pairwise attributions are better suited to RC than token-level attributions across these different RC settings, with our best performance coming from a modification that we propose to an existing pairwise attribution method.