Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models
This work addresses adversarial robustness in vision-language models, which is crucial for reliable AI applications, but it is incremental as it builds on existing adversarial prompt tuning approaches.
The paper tackles the vulnerability of vision-language models to adversarial examples by proposing ER-APT, a method that combines gradient-based and genetic evolution techniques to generate diverse adversarial examples for prompt tuning, resulting in improved robustness and outperforming state-of-the-art methods on benchmark datasets.
Large pre-trained vision-language models (VLMs), such as CLIP, demonstrate impressive generalization but remain highly vulnerable to adversarial examples (AEs). Previous work has explored robust text prompts through adversarial training, achieving some improvement in both robustness and generalization. However, they primarily rely on singlegradient direction perturbations (e.g., PGD) to generate AEs, which lack diversity, resulting in limited improvement in adversarial robustness. To address these limitations, we propose an evolution-based region adversarial prompt tuning method called ER-APT, which combines gradient methods with genetic evolution to generate more diverse and challenging AEs. In each training iteration, we first generate AEs using traditional gradient-based methods. Subsequently, a genetic evolution mechanism incorporating selection, mutation, and crossover is applied to optimize the AEs, ensuring a broader and more aggressive perturbation distribution.The final evolved AEs are used for prompt tuning, achieving region-based adversarial optimization instead of conventional single-point adversarial prompt tuning. We also propose a dynamic loss weighting method to adjust prompt learning efficiency for accuracy and robustness. Experimental evaluations on various benchmark datasets demonstrate the superiority of our proposed method, outperforming stateof-the-art APT methods. The code is released at https://github.com/jiaxiaojunQAQ/ER-APT.