Transformer-based interpretable multi-modal data fusion for skin lesion classification
This addresses the need for interpretable clinical decision support systems in dermatology, though it is incremental as it adapts existing transformer methods to a specific domain.
The paper tackled the problem of limited transparency and trust in deep learning for skin lesion classification by enabling single-stage multi-modal data fusion using transformer-based architectures, resulting in a method that beats state-of-the-art single- and multi-modal DL architectures in image-rich and patient-data-rich environments.
A lot of deep learning (DL) research these days is mainly focused on improving quantitative metrics regardless of other factors. In human-centered applications, like skin lesion classification in dermatology, DL-driven clinical decision support systems are still in their infancy due to the limited transparency of their decision-making process. Moreover, the lack of procedures that can explain the behavior of trained DL algorithms leads to almost no trust from clinical physicians. To diagnose skin lesions, dermatologists rely on visual assessment of the disease and the data gathered from the patient's anamnesis. Data-driven algorithms dealing with multi-modal data are limited by the separation of feature-level and decision-level fusion procedures required by convolutional architectures. To address this issue, we enable single-stage multi-modal data fusion via the attention mechanism of transformer-based architectures to aid in diagnosing skin diseases. Our method beats other state-of-the-art single- and multi-modal DL architectures in image-rich and patient-data-rich environments. Additionally, the choice of the architecture enables native interpretability support for the classification task both in the image and metadata domain with no additional modifications necessary.