DiMPLe -- Disentangled Multi-Modal Prompt Learning: Enhancing Out-Of-Distribution Alignment with Invariant and Spurious Feature Separation
This addresses the issue of poor generalization to novel classes and distribution shifts in multi-modal learning for researchers and practitioners, representing a novel method for a known bottleneck.
The paper tackled the problem of spurious correlations hindering out-of-distribution performance in multi-modal learning by introducing DiMPLe, which disentangles invariant and spurious features across vision and language modalities, resulting in absolute gains of 15.27 in base class accuracy and 44.31 in novel class accuracy.
We introduce DiMPLe (Disentangled Multi-Modal Prompt Learning), a novel approach to disentangle invariant and spurious features across vision and language modalities in multi-modal learning. Spurious correlations in visual data often hinder out-of-distribution (OOD) performance. Unlike prior methods focusing solely on image features, DiMPLe disentangles features within and across modalities while maintaining consistent alignment, enabling better generalization to novel classes and robustness to distribution shifts. Our method combines three key objectives: (1) mutual information minimization between invariant and spurious features, (2) spurious feature regularization, and (3) contrastive learning on invariant features. Extensive experiments demonstrate DiMPLe demonstrates superior performance compared to CoOp-OOD, when averaged across 11 diverse datasets, and achieves absolute gains of 15.27 in base class accuracy and 44.31 in novel class accuracy.