MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning
This addresses the practical issue of incomplete multimodal data for researchers and practitioners in AI, though it is incremental as it builds on existing distillation and regularization techniques.
The paper tackles the problem of missing modality data in multimodal learning, which causes severe performance degradation, and proposes MMANet, a framework that uses margin-aware distillation and modality-aware regularization to improve performance, achieving significant outperformance over state-of-the-art methods in multimodal classification and segmentation tasks.
Multimodal learning has shown great potentials in numerous scenes and attracts increasing interest recently. However, it often encounters the problem of missing modality data and thus suffers severe performance degradation in practice. To this end, we propose a general framework called MMANet to assist incomplete multimodal learning. It consists of three components: the deployment network used for inference, the teacher network transferring comprehensive multimodal information to the deployment network, and the regularization network guiding the deployment network to balance weak modality combinations. Specifically, we propose a novel margin-aware distillation (MAD) to assist the information transfer by weighing the sample contribution with the classification uncertainty. This encourages the deployment network to focus on the samples near decision boundaries and acquire the refined inter-class margin. Besides, we design a modality-aware regularization (MAR) algorithm to mine the weak modality combinations and guide the regularization network to calculate prediction loss for them. This forces the deployment network to improve its representation ability for the weak modality combinations adaptively. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that our MMANet outperforms the state-of-the-art significantly. Code is available at: https://github.com/shicaiwei123/MMANet