Why Retrieval-Augmented Generation Fails: A Graph Perspective
For researchers and practitioners using RAG, this work provides a mechanistic understanding of RAG failures and a practical method to detect and mitigate them.
The paper investigates why retrieval-augmented generation (RAG) systems fail by constructing attribution graphs that model information flow in transformers. It finds that correct predictions have deeper, more distributed evidence flow, while failures show shallow, fragmented flow, and uses these insights to develop a graph-based error detection framework and targeted interventions that reduce errors.
Retrieval-Augmented Generation (RAG) has become a powerful and widely used approach for improving large language models by grounding generation in retrieved evidence. However, RAG systems still produce incorrect answers in many cases. Why RAG fails despite having access to external information remains poorly understood. We present a model-internal study of retrieval-augmented generation that examines how retrieved evidence influences answer generation. Using circuit tracing, we construct attribution graphs that model the flow of information through transformer layers during decoding. These graphs represent interactions among retrieved context, intermediate model activations, and generated tokens, providing a graph, circuit-level view of how external evidence is integrated into the model's reasoning process across multiple question answering benchmarks, we observe consistent structural differences: correct predictions exhibit deeper reasoning paths, more distributed evidence flow, and a more structured pattern of local connectivity, while failed predictions show shallower, fragmented, and overly concentrated evidence flow. Building on these findings, we develop a graph-based error detection framework that uses attribution-graph topology features. Furthermore, we show that attribution graphs enable targeted interventions. By reinforcing question-constrained evidence grounding, we reshape internal routing so that answer generation remains guided by the question, leading to more effective integration of retrieved information and fewer errors.