Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition
This addresses the challenge of improving generalization in few-shot adaptation of VLMs for vision-language recognition, though it is incremental as it builds on existing attribute-based methods.
The paper tackles the problem of Vision-Language Models (VLMs) over-relying on spuriously correlated attributes, which harms out-of-distribution generalization, and proposes methods that significantly enhance accuracy on distribution shifts across 11 datasets and 3 tasks, establishing a new state-of-the-art benchmark.
Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. However, this study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose Spurious Attribute Probing (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-based methods; 2) We introduce Spurious Attribute Shielding (SAS), a plug-and-play module that mitigates the influence of these attributes on prediction, seamlessly integrating into various Parameter-Efficient Fine-Tuning (PEFT) methods. In experiments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance, establishing a new state-of-the-art benchmark.