Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
This work exposes a critical vulnerability in multimodal RAG systems for developers and users, highlighting that metadata-only poisoning can severely disrupt system behaviors.
This paper introduces MM-MEPA, a stealth poisoning attack on multimodal Retrieval-Augmented Generation (RAG) systems that manipulates only the metadata of image-text entries, leaving visual content untouched. The attack successfully steers multimodal retrieval and induces attacker-desired model responses, achieving an attack success rate of up to 91% across various retrievers and multimodal generators.
Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for enhancing multimodal large language models by grounding their responses in external, factual knowledge and thus mitigating hallucinations. However, the integration of externally sourced knowledge bases introduces a critical attack surface. Adversaries can inject malicious multimodal content capable of influencing both retrieval and downstream generation. In this work, we present MM-MEPA, a multimodal poisoning attack that targets the metadata components of image-text entries while leaving the associated visual content unaltered. By only manipulating the metadata, MM-MEPA can still steer multimodal retrieval and induce attacker-desired model responses. We evaluate the attack across multiple benchmark settings and demonstrate its severity. MM-MEPA achieves an attack success rate of up to 91\% consistently disrupting system behaviors across four retrievers and two multimodal generators. Additionally, we assess representative defense strategies and find them largely ineffective against this form of metadata-only poisoning. Our findings expose a critical vulnerability in multimodal RAG and underscore the urgent need for more robust, defense-aware retrieval and knowledge integration methods.