Clinicians' Voice: Fundamental Considerations for XAI in Healthcare
This work addresses the problem of limited practical value in XAI for healthcare by incorporating clinician feedback, though it is incremental as it builds on existing user-centered approaches.
The study tackled the lack of end-user input in explainable AI (XAI) research for healthcare by conducting interviews with clinicians to gather their perspectives, identifying concerns about workflow integration and clinician-patient relations, and highlighting training as a crucial factor for success.
Explainable AI (XAI) holds the promise of advancing the implementation and adoption of AI-based tools in practice, especially in high-stakes environments like healthcare. However, most of the current research lacks input from end users, and therefore their practical value is limited. To address this, we conducted semi-structured interviews with clinicians to discuss their thoughts, hopes, and concerns. Clinicians from our sample generally think positively about developing AI-based tools for clinical practice, but they have concerns about how these will fit into their workflow and how it will impact clinician-patient relations. We further identify training of clinicians on AI as a crucial factor for the success of AI in healthcare and highlight aspects clinicians are looking for in (X)AI-based tools. In contrast to other studies, we take on a holistic and exploratory perspective to identify general requirements for (X)AI products for healthcare before moving on to testing specific tools.