Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
This addresses a fundamental constraint in vision-language models for fine-grained visual recognition, though it is an incremental improvement over existing prompt engineering methods.
The paper tackles CLIP's bias toward global image patterns, which hinders its ability to process localized visual details, and proposes a decomposition and description approach using stochastic multi-crop augmentation to recalibrate attention, achieving promising performance in zero-shot, few-shot, and test-time adaptation settings.
Vision-Language Models (VLMs) like CLIP achieve cross-modal semantic alignment through contrastive learning, exhibiting robust zero-shot generalization. Traditional prompt engineering, however, predominantly relies on coarse-grained category labels, neglecting fine-grained local semantics. Existing approaches assume that VLMs inherently recognize localized visual details and attempt to enhance classification by augmenting text prompts with attribute descriptors generated by large language models. However, our systematic experiments reveal critical limitations: CLIP's strong bias toward global image patterns hinders its ability to process localized visual descriptors. To address this fundamental constraint, we propose a simple, effective, and plug-and-play solution that enables CLIP to ``See Both the Forest and the Trees." Specifically, we employ stochastic multi-crop augmentation to activate CLIP's latent capacity for localized feature analysis. By cropping only partial regions, the approach effectively constrains the model's receptive field and recalibrates its attention mechanism, thereby mitigating its inherent bias. We evaluate the proposed method under zero-shot, few-shot, and test-time adaptation settings, and extensive experiments demonstrate that D&D achieves promising performance.