"Explanation" is Not a Technical Term: The Problem of Ambiguity in XAI
This work tackles a foundational problem in XAI for researchers and practitioners by clarifying explanation requirements to improve system trustworthiness, though it is incremental in refining existing concepts.
The paper addresses the ambiguity in defining 'explanation' within explainable AI (XAI), highlighting a disconnect between system-generated explanations and user needs, and proposes focusing on functional roles, user knowledge, and information availability to evaluate utility while warning against misplaced trust.
There is broad agreement that Artificial Intelligence (AI) systems, particularly those using Machine Learning (ML), should be able to "explain" their behavior. Unfortunately, there is little agreement as to what constitutes an "explanation." This has caused a disconnect between the explanations that systems produce in service of explainable Artificial Intelligence (XAI) and those explanations that users and other audiences actually need, which should be defined by the full spectrum of functional roles, audiences, and capabilities for explanation. In this paper, we explore the features of explanations and how to use those features in evaluating their utility. We focus on the requirements for explanations defined by their functional role, the knowledge states of users who are trying to understand them, and the availability of the information needed to generate them. Further, we discuss the risk of XAI enabling trust in systems without establishing their trustworthiness and define a critical next step for the field of XAI to establish metrics to guide and ground the utility of system-generated explanations.