Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
This addresses the issue of unintended harms in XAI systems for researchers, designers, and organizations, though it is incremental as it builds on existing concepts like dark patterns.
The paper tackles the problem of unanticipated negative effects in Explainable AI (XAI) by introducing explainability pitfalls, which are harmful downstream consequences that occur even without malicious intent, and proposes strategies to address them at research, design, and organizational levels.
To make Explainable AI (XAI) systems trustworthy, understanding harmful effects is just as important as producing well-designed explanations. In this paper, we address an important yet unarticulated type of negative effect in XAI. We introduce explainability pitfalls(EPs), unanticipated negative downstream effects from AI explanations manifesting even when there is no intention to manipulate users. EPs are different from, yet related to, dark patterns, which are intentionally deceptive practices. We articulate the concept of EPs by demarcating it from dark patterns and highlighting the challenges arising from uncertainties around pitfalls. We situate and operationalize the concept using a case study that showcases how, despite best intentions, unsuspecting negative effects such as unwarranted trust in numerical explanations can emerge. We propose proactive and preventative strategies to address EPs at three interconnected levels: research, design, and organizational.