MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition
This addresses the challenge of missing modalities in real-world visual recognition for applications with privacy or resource constraints, representing an incremental improvement over existing prompt learning techniques.
The paper tackles the problem of missing modalities in visual recognition by introducing MoRA, a parameter-efficient fine-tuning method that models cross-modal interactions, achieving an average performance improvement of 5.24% in missing-modality scenarios while using only 25.90% of the inference time and 0.11% of trainable parameters compared to baselines.
Pre-trained vision language models have shown remarkable performance on visual recognition tasks, but they typically assume the availability of complete multimodal inputs during both training and inference. In real-world scenarios, however, modalities may be missing due to privacy constraints, collection difficulties, or resource limitations. While previous approaches have addressed this challenge using prompt learning techniques, they fail to capture the cross-modal relationships necessary for effective multimodal visual recognition and suffer from inevitable computational overhead. In this paper, we introduce MoRA, a parameter-efficient fine-tuning method that explicitly models cross-modal interactions while maintaining modality-specific adaptations. MoRA introduces modality-common parameters between text and vision encoders, enabling bidirectional knowledge transfer. Additionally, combined with the modality-specific parameters, MoRA allows the backbone model to maintain inter-modality interaction and enable intra-modality flexibility. Extensive experiments on standard benchmarks demonstrate that MoRA achieves an average performance improvement in missing-modality scenarios by 5.24% and uses only 25.90% of the inference time compared to the SOTA method while requiring only 0.11% of trainable parameters compared to full fine-tuning.