VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts
This addresses safety vulnerabilities in VLMs for secure multimodal AI deployment, representing an incremental improvement over existing defenses.
The paper tackles the problem of malicious prompt attacks on Vision-Language Models (VLMs) by proposing VLMShield, a lightweight safety detector that efficiently identifies such attacks, demonstrating superior performance in robustness, efficiency, and utility.
Vision-Language Models (VLMs) face significant safety vulnerabilities from malicious prompt attacks due to weakened alignment during visual integration. Existing defenses suffer from efficiency and robustness. To address these challenges, we first propose the Multimodal Aggregated Feature Extraction (MAFE) framework that enables CLIP to handle long text and fuse multimodal information into unified representations. Through empirical analysis of MAFE-extracted features, we discover distinct distributional patterns between benign and malicious prompts. Building upon this finding, we develop VLMShield, a lightweight safety detector that efficiently identifies multimodal malicious attacks as a plug-and-play solution. Extensive experiments demonstrate superior performance across multiple dimensions, including robustness, efficiency, and utility. Through our work, we hope to pave the way for more secure multimodal AI deployment. Code is available at [this https URL](https://github.com/pgqihere/VLMShield).