Zero-Shot Robustness of Vision Language Models Via Confidence-Aware Weighting
This work addresses the robustness issue in vision-language models for zero-shot applications, offering an incremental improvement over existing methods.
The paper tackles the problem of vision-language models being vulnerable to adversarial attacks in zero-shot settings by proposing Confidence-Aware Weighting (CAW), which improves both clean and robust accuracy without sacrificing generalization, as shown by outperforming recent methods like PMG-AFT and TGA-ZSR under strong attacks on multiple datasets.
Vision-language models like CLIP demonstrate impressive zero-shot generalization but remain highly vulnerable to adversarial attacks. In this work, we propose Confidence-Aware Weighting (CAW) to enhance zero-shot robustness in vision-language models. CAW consists of two components: (1) a Confidence-Aware loss that prioritizes uncertain adversarial examples by scaling the KL divergence between clean and adversarial predictions, and (2) a feature alignment regularization that preserves semantic consistency by minimizing the distance between frozen and fine-tuned image encoder features on adversarial inputs. These components work jointly to improve both clean and robust accuracy without sacrificing generalization. Extensive experiments on TinyImageNet and 14 additional datasets show that CAW outperforms recent methods such as PMG-AFT and TGA-ZSR under strong attacks like AutoAttack, while using less memory.