"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
This work addresses a gap in understanding how explainability can support human-AI interaction for end-users, though it is incremental as it builds on existing XAI research.
The study investigated end-users' explainability needs in a real-world AI application, finding that participants prioritize practically useful information for collaboration over technical details and prefer part-based explanations resembling human reasoning.
Despite the proliferation of explainable AI (XAI) methods, little is understood about end-users' explainability needs and behaviors around XAI explanations. To address this gap and contribute to understanding how explainability can support human-AI interaction, we conducted a mixed-methods study with 20 end-users of a real-world AI application, the Merlin bird identification app, and inquired about their XAI needs, uses, and perceptions. We found that participants desire practically useful information that can improve their collaboration with the AI, more so than technical system details. Relatedly, participants intended to use XAI explanations for various purposes beyond understanding the AI's outputs: calibrating trust, improving their task skills, changing their behavior to supply better inputs to the AI, and giving constructive feedback to developers. Finally, among existing XAI approaches, participants preferred part-based explanations that resemble human reasoning and explanations. We discuss the implications of our findings and provide recommendations for future XAI design.