Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration
This addresses a critical problem for designers and users of AI systems by highlighting risks in human-AI interaction, though it is incremental as it builds on prior XAI research.
The study investigated how incorrect explanations from explainable AI (XAI) systems affect human-AI collaboration, finding that they cause a misinformation effect that impairs human task execution and team performance, with empirical evidence from a study involving 160 participants.
Across various applications, humans increasingly use black-box artificial intelligence (AI) systems without insight into these systems' reasoning. To counter this opacity, explainable AI (XAI) methods promise enhanced transparency and interpretability. While recent studies have explored how XAI affects human-AI collaboration, few have examined the potential pitfalls caused by incorrect explanations. The implications for humans can be far-reaching but have not been explored extensively. To investigate this, we ran a study (n=160) on AI-assisted decision-making in which humans were supported by XAI. Our findings reveal a misinformation effect when incorrect explanations accompany correct AI advice with implications post-collaboration. This effect causes humans to infer flawed reasoning strategies, hindering task execution and demonstrating impaired procedural knowledge. Additionally, incorrect explanations compromise human-AI team-performance during collaboration. With our work, we contribute to HCI by providing empirical evidence for the negative consequences of incorrect explanations on humans post-collaboration and outlining guidelines for designers of AI.