Distributionally Robust Multimodal Machine Learning
It addresses robustness for multimodal models in high-stakes applications, but appears incremental as it builds on existing DRO methods.
The paper tackles the problem of distributionally robust multimodal machine learning by proposing a novel DRO framework, which improves robustness in simulations and real-world datasets with theoretical guarantees.
We consider the problem of distributionally robust multimodal machine learning. Existing approaches often rely on merging modalities on the feature level (early fusion) or heuristic uncertainty modeling, which downplays modality-aware effects and provide limited insights. We propose a novel distributionally robust optimization (DRO) framework that aims to study both the theoretical and practical insights of multimodal machine learning. We first justify this setup and show the significance of this problem through complexity analysis. We then establish both generalization upper bounds and minimax lower bounds which provide performance guarantees. These results are further extended in settings where we consider encoder-specific error propogations. Empirically, we demonstrate that our approach improves robustness in both simulation settings and real-world datasets. Together, these findings provide a principled foundation for employing multimodal machine learning models in high-stakes applications where uncertainty is unavoidable.