What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
This work addresses the robustness-accuracy trade-off for VLMs, offering a practical solution to enhance adversarial robustness without compromising clean accuracy, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the trade-off between adversarial robustness and clean accuracy in Vision-Language Models (VLMs) by analyzing how robustness mechanisms function internally, finding that robustness is localized in shallow layers due to low-frequency spectral bias and input-insensitive attention patterns. The result is R-Adapt, a framework that freezes pre-trained weights and adapts only initial layers, achieving state-of-the-art performance on 18 datasets and generalizing to large VLMs like LLaVA and Qwen-VL.
Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.