From Explainable to Explained AI: Ideas for Falsifying and Quantifying Explanations
This work addresses the need for reliable explanations in clinical integration of AI systems, particularly in digital pathology, though it appears incremental as it builds on existing explainability techniques.
The paper tackles the problem of moving from explainable to explained AI in medical image analysis by proposing a human-machine-VLM interaction system for computational pathology, which allows qualitative testing and quantitative comparison of explanations, demonstrating the ability to distinguish competing explanations.
Explaining deep learning models is essential for clinical integration of medical image analysis systems. A good explanation highlights if a model depends on spurious features that undermines generalization and harms a subset of patients or, conversely, may present novel biological insights. Although techniques like GradCAM can identify influential features, they are measurement tools that do not themselves form an explanation. We propose a human-machine-VLM interaction system tailored to explaining classifiers in computational pathology, including multi-instance learning for whole-slide images. Our proof of concept comprises (1) an AI-integrated slide viewer to run sliding-window experiments to test claims of an explanation, and (2) quantification of an explanation's predictiveness using general-purpose vision-language models. The results demonstrate that this allows us to qualitatively test claims of explanations and can quantifiably distinguish competing explanations. This offers a practical path from explainable AI to explained AI in digital pathology and beyond. Code and prompts are available at https://github.com/nki-ai/x2x.