Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models
This addresses security concerns for real-time online applications using vision-language models, though it is incremental as it builds on existing universal adversarial perturbation methods.
The paper tackled the vulnerability of vision-language pre-training models to adversarial attacks by proposing DO-UAP, a direct optimization-based universal adversarial perturbation method that reduces time consumption by 23-fold while improving attack performance.
Vision-language pre-training (VLP) models, trained on large-scale image-text pairs, have become widely used across a variety of downstream vision-and-language (V+L) tasks. This widespread adoption raises concerns about their vulnerability to adversarial attacks. Non-universal adversarial attacks, while effective, are often impractical for real-time online applications due to their high computational demands per data instance. Recently, universal adversarial perturbations (UAPs) have been introduced as a solution, but existing generator-based UAP methods are significantly time-consuming. To overcome the limitation, we propose a direct optimization-based UAP approach, termed DO-UAP, which significantly reduces resource consumption while maintaining high attack performance. Specifically, we explore the necessity of multimodal loss design and introduce a useful data augmentation strategy. Extensive experiments conducted on three benchmark VLP datasets, six popular VLP models, and three classical downstream tasks demonstrate the efficiency and effectiveness of DO-UAP. Specifically, our approach drastically decreases the time consumption by 23-fold while achieving a better attack performance.