Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention
This addresses a security vulnerability in RAG systems for users relying on up-to-date and accurate LLM responses, offering a novel defense method against poisoning attacks.
The paper tackled the problem of corpus knowledge poisoning attacks on Retrieval Augmented Generation (RAG) by introducing Sparse Document Attention RAG (SDAG), which uses a block-sparse attention mechanism to disallow cross-attention between retrieved documents, resulting in substantially outperforming standard causal attention in attack success rate and achieving statistically significant improvements when integrated with state-of-the-art defenses.
Retrieval Augmented Generation (RAG) is a highly effective paradigm for keeping LLM-based responses up-to-date and reducing the likelihood of hallucinations. Yet, RAG was recently shown to be quite vulnerable to corpus knowledge poisoning: an attacker injects misleading documents to the corpus to steer an LLM's output to an undesired response. We argue that the standard causal attention mechanism in LLMs enables harmful cross-document interactions, specifically in cases of attacks. Accordingly, we introduce a novel defense approach for RAG: Sparse Document Attention RAG (SDAG). This is a block-sparse attention mechanism that disallows cross-attention between retrieved documents. SDAG requires a minimal inference-time change to the attention mask; furthermore, no fine-tuning or additional architectural changes are needed. We present an empirical evaluation of LLM-based question answering (QA) with a variety of attack strategies on RAG. We show that our SDAG method substantially outperforms the standard causal attention mechanism in terms of attack success rate. We further demonstrate the clear merits of integrating SDAG with state-of-the-art RAG defense methods. Specifically, the integration results in performance that is statistically significantly better than the state-of-the-art.