Explainable AI for Clinical Outcome Prediction: A Survey of Clinician Perceptions and Preferences
This work addresses the need for explainable AI in clinical settings to help clinicians integrate AI predictions into decision-making, but it is incremental as it focuses on surveying preferences rather than developing new methods.
The study surveyed 32 clinicians to assess their preferences among four explainable AI techniques for interpreting predictions from an ICU mortality model using EHR data, resulting in recommendations for when each technique is appropriate and its limitations.
Explainable AI (XAI) techniques are necessary to help clinicians make sense of AI predictions and integrate predictions into their decision-making workflow. In this work, we conduct a survey study to understand clinician preference among different XAI techniques when they are used to interpret model predictions over text-based EHR data. We implement four XAI techniques (LIME, Attention-based span highlights, exemplar patient retrieval, and free-text rationales generated by LLMs) on an outcome prediction model that uses ICU admission notes to predict a patient's likelihood of experiencing in-hospital mortality. Using these XAI implementations, we design and conduct a survey study of 32 practicing clinicians, collecting their feedback and preferences on the four techniques. We synthesize our findings into a set of recommendations describing when each of the XAI techniques may be more appropriate, their potential limitations, as well as recommendations for improvement.