Evolution of Low-Level and Texture Human-CLIP Alignment
This provides insights into optimizing the trade-off between perceptual alignment and robustness in vision-language models, though it is incremental as it builds on existing CLIP observations.
The study investigated why CLIP's alignment with low-level human image quality assessments peaks early in training and then declines, finding that initial learning focuses on low-level features and texture bias, which later shifts to shape-based representations for better noise robustness.
During the training of multi-modal models like CLIP, we observed an intriguing phenomenon: the correlation with low-level human image quality assessments peaks in the early epochs before gradually declining. This study investigates this observation and seeks to understand its causes through two key factors: shape-texture bias alignment and classification accuracy drop under noise. Our findings suggest that CLIP initially learn low-level visual features, enhancing its alignment with low-level human perception but also increasing its sensitivity to noise and its texture bias. As training progresses, the model shifts toward more abstract shape-based representations, improving noise robustness but reducing alignment with low-level human perception. These results suggest that these factors shared an underlying learning mechanism and provide new insights into optimizing the trade-off between perceptual alignment and robustness in vision-language models.