Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models
This addresses the challenge of incomplete multimodal data in real-world applications, which hinders the use and robustness of pretrained models, though it is incremental as it builds on existing unimodal and joint-embedding techniques.
The paper tackles the problem of missing modalities in multimodal learning by proposing a framework that integrates parameter-efficient fine-tuning of unimodal models with self-supervised joint-embedding learning to predict missing modality embeddings during inference, demonstrating effectiveness and robustness on benchmark datasets.
Multimodal learning typically relies on the assumption that all modalities are fully available during both the training and inference phases. However, in real-world scenarios, consistently acquiring complete multimodal data presents significant challenges due to various factors. This often leads to the issue of missing modalities, where data for certain modalities are absent, posing considerable obstacles not only for the availability of multimodal pretrained models but also for their fine-tuning and the preservation of robustness in downstream tasks. To address these challenges, we propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method. This framework enables the model to predict the embedding of a missing modality in the representation space during inference. Our method effectively predicts the missing embedding through prompt tuning, leveraging information from available modalities. We evaluate our approach on several multimodal benchmark datasets and demonstrate its effectiveness and robustness across various scenarios of missing modalities.