Towards Robust Multimodal Prompting With Missing Modalities
This addresses robustness issues in multimodal AI systems for applications like vision-language tasks, but it is incremental as it builds on existing prompting methods.
The paper tackles the exponential growth of prompts and lack of robustness in multimodal prompting with missing modalities by proposing a simple prompt design using modality-specific tokens and orthogonality, resulting in enhanced performance and robustness while reducing prompt count.
Recently, multimodal prompting, which introduces learnable missing-aware prompts for all missing modality cases, has exhibited impressive performance. However, it encounters two critical issues: 1) The number of prompts grows exponentially as the number of modalities increases; and 2) It lacks robustness in scenarios with different missing modality settings between training and inference. In this paper, we propose a simple yet effective prompt design to address these challenges. Instead of using missing-aware prompts, we utilize prompts as modality-specific tokens, enabling them to capture the unique characteristics of each modality. Furthermore, our prompt design leverages orthogonality between prompts as a key element to learn distinct information across different modalities and promote diversity in the learned representations. Extensive experiments demonstrate that our prompt design enhances both performance and robustness while reducing the number of prompts.