UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models
This addresses a critical safety problem for users of MLLMs by providing a universal defense mechanism, though it is incremental as it builds on existing safety methods.
The paper tackles the vulnerability of multimodal large language models (MLLMs) to jailbreak attacks by proposing UniGuard, a safety guardrail that reduces harmful responses, achieving broad defense across multiple models and attack strategies with minimal computational cost.
Multimodal large language models (MLLMs) have revolutionized vision-language understanding but remain vulnerable to multimodal jailbreak attacks, where adversarial inputs are meticulously crafted to elicit harmful or inappropriate responses. We propose UniGuard, a novel multimodal safety guardrail that jointly considers the unimodal and cross-modal harmful signals. UniGuard trains a multimodal guardrail to minimize the likelihood of generating harmful responses in a toxic corpus. The guardrail can be seamlessly applied to any input prompt during inference with minimal computational costs. Extensive experiments demonstrate the generalizability of UniGuard across multiple modalities, attack strategies, and multiple state-of-the-art MLLMs, including LLaVA, Gemini Pro, GPT-4o, MiniGPT-4, and InstructBLIP. Notably, this robust defense mechanism maintains the models' overall vision-language understanding capabilities.