Explaining the Road Not Taken
It addresses a gap in explainable AI for NLP users, but is incremental as it synthesizes existing work without proposing new methods.
The paper analyzed over 200 NLP papers to compare common explanation methods (e.g., feature attribution) against user needs from the XAI Question Bank, finding that most interpretations fail to explain why models choose one result over similar alternatives.
It is unclear if existing interpretations of deep neural network models respond effectively to the needs of users. This paper summarizes the common forms of explanations (such as feature attribution, decision rules, or probes) used in over 200 recent papers about natural language processing (NLP), and compares them against user questions collected in the XAI Question Bank. We found that although users are interested in explanations for the road not taken -- namely, why the model chose one result and not a well-defined, seemly similar legitimate counterpart -- most model interpretations cannot answer these questions.