Simple Methods Defend RAG Systems Well Against Real-World Attacks
This work addresses safety-critical applications by defending RAG systems against real-world attacks, though it is incremental as it builds on existing OOD detection methods.
The paper tackled the problem of ensuring safety and relevance in Retrieval-Augmented Generation (RAG) systems by evaluating four Out-Of-Domain (OOD) query detection methods, including PCA and Neural Collapse adaptations, and validated them on datasets like StackExchange and real-world applications such as a COVID-19 vaccine chatbot, showing that an external OOD detector is crucial for maintaining response relevance.
Ensuring safety and in-domain responses for Retrieval-Augmented Generation (RAG) systems is paramount in safety-critical applications, yet remains a significant challenge. To address this, we evaluate four methodologies for Out-Of-Domain (OOD) query detection: GPT-4o, regression-based, Principal Component Analysis (PCA)-based, and Neural Collapse (NC), to ensure the RAG system only responds to queries confined to the system's knowledge base. Specifically, our evaluation explores two novel dimensionality reduction and feature separation strategies: \textit{PCA}, where top components are selected using explained variance or OOD separability, and an adaptation of \textit{Neural Collapse Feature Separation}. We validate our approach on standard datasets (StackExchange and MSMARCO) and real-world applications (Substance Use and COVID-19), including tests against LLM-simulated and actual attacks on a COVID-19 vaccine chatbot. Through human and LLM-based evaluations of response correctness and relevance, we confirm that an external OOD detector is crucial for maintaining response relevance.