Dynamic Multimodal Information Bottleneck for Multimodality Classification
This work addresses the challenge of handling redundant and noisy multimodal data in AI-based medical diagnosis, which is crucial for improving reliability in clinical applications, though it is incremental as it builds on existing multimodal fusion techniques.
The paper tackles the problem of robust multimodal classification in clinical settings by proposing a dynamic multimodal information bottleneck framework that filters out task-irrelevant information and noise while preserving task-relevant features, achieving state-of-the-art performance and significant robustness, as it remains effective even with large-scale noisy channels.
Effectively leveraging multimodal data such as various images, laboratory tests and clinical information is gaining traction in a variety of AI-based medical diagnosis and prognosis tasks. Most existing multi-modal techniques only focus on enhancing their performance by leveraging the differences or shared features from various modalities and fusing feature across different modalities. These approaches are generally not optimal for clinical settings, which pose the additional challenges of limited training data, as well as being rife with redundant data or noisy modality channels, leading to subpar performance. To address this gap, we study the robustness of existing methods to data redundancy and noise and propose a generalized dynamic multimodal information bottleneck framework for attaining a robust fused feature representation. Specifically, our information bottleneck module serves to filter out the task-irrelevant information and noises in the fused feature, and we further introduce a sufficiency loss to prevent dropping of task-relevant information, thus explicitly preserving the sufficiency of prediction information in the distilled feature. We validate our model on an in-house and a public COVID19 dataset for mortality prediction as well as two public biomedical datasets for diagnostic tasks. Extensive experiments show that our method surpasses the state-of-the-art and is significantly more robust, being the only method to remain performance when large-scale noisy channels exist. Our code is publicly available at https://github.com/ayanglab/DMIB.