Re-M3Dr: Rebalanced MultiModal Mean Deviation Regression
Provides a practical solution for improving multimodal regression in medical imaging, specifically for ophthalmologists assessing visual field loss.
Multimodal fusion of OCT and fundus photography for predicting mean deviation in ophthalmology initially underperforms unimodal models due to coupled data-modality imbalance. The proposed Re-M3Dr framework, using adaptive contrastive learning and sharpness-aware gradient modulation, achieves a 29% average reduction in MSE over state-of-the-art multimodal methods.
Mean Deviation (MD) is a critical metric for assessing visual field loss in ophthalmology. While previous work has focused solely on predicting MD from Optical Coherence Tomography (OCT), it is intuitive to assume that combining OCT with another imaging of fundus photography (FP) could improve performance, as two ophthalmic medical imaging provide complementary information. This is particularly expected when sophisticated multi-objective optimization is applied, as documented in common multimodal classification. Surprisingly, our investigations reveal that multimodal fusion in this medical imaging scenario performs worse than unimodal model. Through detailed analysis, we identify the root cause as a coupled imbalance between data distribution and modality learning conflict. This imbalance distorts the optimization landscape, leading to unstable training. To address this challenge, we propose the method of Rebalanced MultiModal Mean Deviation Regression (Re-M3Dr), a novel multimodal regression framework. We enhance unimodal representation through adaptive margin based supervised contrastive learning. Then, our framework stabilizes the joint optimization with the sharpness-aware gradient modulation. Experimental results on both public and private clinical datasets show average 29\% reduction in MSE compared to SOTA multimodal learning methods, demonstrating the superiority of Re-M3Dr. The code is available in the supplementary materials.