Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation
This addresses the challenge of computationally intensive context attribution in RAG systems, which is crucial for enhancing the accuracy and reliability of generated responses in applications like question answering.
The paper tackles the problem of reliably attributing generated content to specific context segments in Retrieval-Augmented Generation (RAG) by introducing a Jensen-Shannon Divergence driven method (ARC-JSD) that enables efficient and accurate identification of essential context sentences without additional fine-tuning, achieving superior accuracy and significant computational efficiency improvements on benchmarks like TyDi QA, Hotpot QA, and Musique.
Retrieval-Augmented Generation (RAG) leverages large language models (LLMs) combined with external contexts to enhance the accuracy and reliability of generated responses. However, reliably attributing generated content to specific context segments, context attribution, remains challenging due to the computationally intensive nature of current methods, which often require extensive fine-tuning or human annotation. In this work, we introduce a novel Jensen-Shannon Divergence driven method to Attribute Response to Context (ARC-JSD), enabling efficient and accurate identification of essential context sentences without additional fine-tuning, gradient-calculation or surrogate modelling. Evaluations on a wide range of RAG benchmarks, such as TyDi QA, Hotpot QA, and Musique, using instruction-tuned LLMs in different scales demonstrate superior accuracy and significant computational efficiency improvements compared to the previous surrogate-based method. Furthermore, our mechanistic analysis reveals specific attention heads and multilayer perceptron (MLP) layers responsible for context attribution, providing valuable insights into the internal workings of RAG models and how they affect RAG behaviours. Our code is available at https://github.com/ruizheliUOA/ARC_JSD.