Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals
This addresses interpretability and reliability issues in multimodal learning for researchers and practitioners, though it is incremental as it builds on existing Shapley values and conformal inference methods.
The paper tackled the problem of quantifying modality importance with uncertainty in multimodal learning, introducing conformal Shapley intervals to provide uncertainty-aware importance intervals and a modality selection procedure with provable optimality, achieving strong predictive performance using only a small number of informative modalities.
Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalities are genuinely informative and to what extent their contributions can be trusted. Quantifying modality level importance together with uncertainty is therefore central to interpretable and reliable multimodal learning. We introduce conformal Shapley intervals, a framework that combines Shapley values with conformal inference to construct uncertainty-aware importance intervals for each modality. Building on these intervals, we propose a modality selection procedure with a provable optimality guarantee: conditional on the observed features, the selected subset of modalities achieves performance close to that of the optimal subset. We demonstrate the effectiveness of our approach on multiple datasets, showing that it provides meaningful uncertainty quantification and strong predictive performance while relying on only a small number of informative modalities.