Multimodal Explainability via Latent Shift applied to COVID-19 stratification
This work addresses the need for explainable AI in healthcare to support diagnosis and treatment decisions, though it is incremental as it builds on existing multimodal and explainability techniques.
The paper tackles the problem of interpreting multimodal AI decisions in healthcare by proposing a deep architecture that jointly learns reconstructions and classifications from tabular and imaging data, using a latent shift method to generate counterfactual explanations and modality importance scores, validated on the AIforCOVID dataset for COVID-19 patient stratification without degrading classification performance.
We are witnessing a widespread adoption of artificial intelligence in healthcare. However, most of the advancements in deep learning in this area consider only unimodal data, neglecting other modalities. Their multimodal interpretation necessary for supporting diagnosis, prognosis and treatment decisions. In this work we present a deep architecture, which jointly learns modality reconstructions and sample classifications using tabular and imaging data. The explanation of the decision taken is computed by applying a latent shift that, simulates a counterfactual prediction revealing the features of each modality that contribute the most to the decision and a quantitative score indicating the modality importance. We validate our approach in the context of COVID-19 pandemic using the AIforCOVID dataset, which contains multimodal data for the early identification of patients at risk of severe outcome. The results show that the proposed method provides meaningful explanations without degrading the classification performance.