LeakDojo: Decoding the Leakage Threats of RAG Systems
For developers and researchers of RAG systems, this work provides a systematic evaluation framework and actionable insights into leakage risks, though it is an incremental contribution building on existing attack methods.
LeakDojo is a framework for evaluating leakage risks in Retrieval-Augmented Generation (RAG) systems. Benchmarking six attacks across fourteen LLMs and four datasets, it finds that query generation and adversarial instructions independently contribute to leakage, stronger instruction-following increases risk, and improvements in RAG faithfulness can heighten leakage.
Retrieval-Augmented Generation (RAG) enables large language models (LLMs) to leverage external knowledge, but also exposes valuable RAG databases to leakage attacks. As RAG systems grow more complex and LLMs exhibit stronger instruction-following capabilities, existing studies fall short of systematically assessing RAG leakage risks. We present LeakDojo, a configurable framework for controlled evaluation of RAG leakage. Using LeakDojo, we benchmark six existing attacks across fourteen LLMs, four datasets, and diverse RAG systems. Our study reveals that (1) query generation and adversarial instructions contribute independently to leakage, with overall leakage well approximated by their product; (2) stronger instruction-following capability correlates with higher leakage risk; and (3) improvements in RAG faithfulness can introduce increased leakage risk. These findings provide actionable insights for understanding and mitigating RAG leakage in practice. Our codebase is available at https://github.com/yeasen-z/LeakDojo.