Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation
This addresses a practical limitation for deploying multimodal AI in scenarios with incomplete data, representing a novel hybrid approach rather than a foundational breakthrough.
The paper tackles the problem of multimodal models struggling with missing modalities in real-world applications by introducing SURE, a framework that reconstructs missing modalities and provides uncertainty estimates, achieving state-of-the-art performance across tasks like sentiment analysis and action recognition.
Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.