Mitigating Cache Noise in Test-Time Adaptation for Large Vision-Language Models
This addresses performance degradation in vision-language models for downstream tasks under distribution shifts, but it is incremental as it builds on existing cache-based TTA methods.
The paper tackled the problem of noisy pseudo-labels in cache-based test-time adaptation for vision-language models, which cause performance degradation under distribution shifts, and introduced the CRG method that improved robustness and adaptability, outperforming state-of-the-art methods on 13 benchmarks.
Test-time adaptation (TTA) of visual language models has recently attracted significant attention as a solution to the performance degradation caused by distribution shifts in downstream tasks. However, existing cache-based TTA methods have certain limitations. They mainly rely on the accuracy of cached feature labels, and the presence of noisy pseudo-labels can cause these features to deviate from their true distribution. This makes cache retrieval methods based on similarity matching highly sensitive to outliers or extreme samples. Moreover, current methods lack effective mechanisms to model class distributions, which limits their ability to fully exploit the potential of cached information. To address these challenges, we introduce a comprehensive and reliable caching mechanism and propose a novel zero-shot TTA method called "Cache, Residual, Gaussian" (CRG). This method not only employs learnable residual parameters to better align positive and negative visual prototypes with text prototypes, thereby optimizing the quality of cached features, but also incorporates Gaussian Discriminant Analysis (GDA) to dynamically model intra-class feature distributions, further mitigating the impact of noisy features. Experimental results on 13 benchmarks demonstrate that CRG outperforms state-of-the-art TTA methods, showcasing exceptional robustness and adaptability.