Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning
This addresses robustness issues for real-world deployments of multi-modal models, particularly for domains where pre-training mitigation is costly, but it is incremental as it builds on existing fine-tuning approaches.
The paper tackles the problem of spurious correlations degrading generalization in multi-modal models by proposing a fine-tuning method that uses multi-modal contrastive loss to separate spurious attributes, resulting in worst-group accuracy improvements of 23% and 32% on CLIP models compared to ERM while maintaining average accuracy.
Spurious correlations that degrade model generalization or lead the model to be right for the wrong reasons are one of the main robustness concerns for real-world deployments. However, mitigating these correlations during pre-training for large-scale models can be costly and impractical, particularly for those without access to high-performance computing resources. This paper proposes a novel approach to address spurious correlations during fine-tuning for a given domain of interest. With a focus on multi-modal models (e.g., CLIP), the proposed method leverages different modalities in these models to detect and explicitly set apart spurious attributes from the affected class, achieved through a multi-modal contrastive loss function that expresses spurious relationships through language. Our experimental results and in-depth visualizations on CLIP show that such an intervention can effectively i) improve the model's accuracy when spurious attributes are not present, and ii) directs the model's activation maps towards the actual class rather than the spurious attribute when present. In particular, on the Waterbirds dataset, our algorithm achieved a worst-group accuracy 23% higher than ERM on CLIP with a ResNet-50 backbone, and 32% higher on CLIP with a ViT backbone, while maintaining the same average accuracy as ERM.