Multimodal Prompting with Missing Modalities for Visual Recognition
This work tackles practical deployment issues in multimodal AI for real-world applications where data or resources may be incomplete, offering an efficient solution.
The paper addresses multimodal learning challenges in visual recognition when modalities are missing during training or testing and computational resources are limited, proposing a prompt learning framework that improves performance under missing-modality cases while using less than 1% learnable parameters compared to full model training.
In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire model. We further explore the effect of different prompt configurations and analyze the robustness to missing modality. Extensive experiments are conducted to show the effectiveness of our prompt learning framework that improves the performance under various missing-modality cases, while alleviating the requirement of heavy model re-training. Code is available.