Robust Multimodal Learning via Cross-Modal Proxy Tokens
This addresses robustness issues in multimodal learning for applications like vision-language tasks, though it is incremental as it builds on existing methods with a novel adaptation.
The paper tackles the problem of performance drop in multimodal models when modalities are missing during inference by proposing cross-modal proxy tokens (CMPTs) to approximate missing modalities, resulting in outperforming state-of-the-art baselines across five datasets with various missing rates while maintaining competitive performance in complete-modality settings.
Multimodal models often experience a significant performance drop when one or more modalities are missing during inference. To address this challenge, we propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTs), which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with a task-specific loss. Extensive experiments on five multimodal datasets show that our method outperforms state-of-the-art baselines across various missing rates while achieving competitive results in complete-modality settings. Overall, our method offers a flexible and efficient solution for robust multimodal learning. The code for this paper is available at: https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens.