On Two XAI Cultures: A Case Study of Non-technical Explanations in Deployed AI System
It tackles the problem of making AI explanations accessible to non-technical stakeholders, which is crucial for real-world deployments, though it is incremental by synthesizing lessons from a case study.
The paper addresses the gap in explainable AI (XAI) for non-technical audiences, highlighting distinct cultures between AI experts and non-experts in deployed systems, and presents a case study where non-technical explanations enabled successful AI deployment in a regulated industry.
Explainable AI (XAI) research has been booming, but the question "$\textbf{To whom}$ are we making AI explainable?" is yet to gain sufficient attention. Not much of XAI is comprehensible to non-AI experts, who nonetheless, are the primary audience and major stakeholders of deployed AI systems in practice. The gap is glaring: what is considered "explained" to AI-experts versus non-experts are very different in practical scenarios. Hence, this gap produced two distinct cultures of expectations, goals, and forms of XAI in real-life AI deployments. We advocate that it is critical to develop XAI methods for non-technical audiences. We then present a real-life case study, where AI experts provided non-technical explanations of AI decisions to non-technical stakeholders, and completed a successful deployment in a highly regulated industry. We then synthesize lessons learned from the case, and share a list of suggestions for AI experts to consider when explaining AI decisions to non-technical stakeholders.