On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces
For researchers and practitioners using vision-language models, this work reveals a new mechanism of adversarial vulnerability and provides a method to evaluate and potentially improve robustness.
The paper proposes a white-box spectral-subspace-guided attack (SSGRA) that exploits the spectral structure of intermediate linear transformations in vision-language models, achieving improved attack effectiveness over existing baselines.
Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.