TrojVLM: Backdoor Attack Against Vision Language Models
This study reveals a critical security vulnerability in VLMs, which are increasingly used for multimodal AI applications, posing risks to systems relying on image-to-text generation.
The paper tackles the problem of backdoor attacks on Vision Language Models (VLMs) for image-to-text generation, introducing TrojVLM which successfully inserts target text into outputs when triggered by poisoned images while preserving original semantics, as confirmed in evaluations on image captioning and VQA tasks.
The emergence of Vision Language Models (VLMs) is a significant advancement in integrating computer vision with Large Language Models (LLMs) to produce detailed text descriptions based on visual inputs, yet it introduces new security vulnerabilities. Unlike prior work that centered on single modalities or classification tasks, this study introduces TrojVLM, the first exploration of backdoor attacks aimed at VLMs engaged in complex image-to-text generation. Specifically, TrojVLM inserts predetermined target text into output text when encountering poisoned images. Moreover, a novel semantic preserving loss is proposed to ensure the semantic integrity of the original image content. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of TrojVLM in maintaining original semantic content while triggering specific target text outputs. This study not only uncovers a critical security risk in VLMs and image-to-text generation but also sets a foundation for future research on securing multimodal models against such sophisticated threats.