Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
This work addresses the critical need for robust VLMs in safety-critical environments, though it appears incremental as it builds on existing methods with novel prompt-based enhancements.
The paper tackled the problem of adversarial robustness in Vision-Language Models against image-based attacks by investigating model design choices and introducing cost-effective prompt formatting techniques, resulting in substantial improvements in robustness against strong attacks like Auto-PGD.
Vision-Language Models (VLMs) have witnessed a surge in both research and real-world applications. However, as they are becoming increasingly prevalent, ensuring their robustness against adversarial attacks is paramount. This work systematically investigates the impact of model design choices on the adversarial robustness of VLMs against image-based attacks. Additionally, we introduce novel, cost-effective approaches to enhance robustness through prompt formatting. By rephrasing questions and suggesting potential adversarial perturbations, we demonstrate substantial improvements in model robustness against strong image-based attacks such as Auto-PGD. Our findings provide important guidelines for developing more robust VLMs, particularly for deployment in safety-critical environments.