Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
This provides a practical, plug-and-play solution for improving robustness in existing vision-language models against adversarial attacks, though it is incremental as it builds on existing defense concepts.
The paper tackles the problem of adversarial attacks on vision-language models by introducing a lightweight defense using tensor decomposition that requires no retraining, restoring up to 12.3% of lost performance on benchmarks like Flickr30K and COCO.
Vision language models (VLMs) excel in multimodal understanding but are prone to adversarial attacks. Existing defenses often demand costly retraining or significant architecture changes. We introduce a lightweight defense using tensor decomposition suitable for any pre-trained VLM, requiring no retraining. By decomposing and reconstructing vision encoder representations, it filters adversarial noise while preserving meaning. Experiments with CLIP on COCO and Flickr30K show improved robustness. On Flickr30K, it restores 12.3\% performance lost to attacks, raising Recall@1 accuracy from 7.5\% to 19.8\%. On COCO, it recovers 8.1\% performance, improving accuracy from 3.8\% to 11.9\%. Analysis shows Tensor Train decomposition with low rank (8-32) and low residual strength ($α=0.1-0.2$) is optimal. This method is a practical, plug-and-play solution with minimal overhead for existing VLMs.