How Do LLMs Cite? A Mechanistic Interpretation of Attribution in Retrieval-Augmented Generation
For researchers and practitioners relying on RAG for trustworthy AI, this work exposes that inline citations may not reflect genuine reasoning, challenging assumptions about verifiability.
This paper provides the first mechanistic explanation of how LLMs decide to cite sources in RAG, revealing a distributed 'attributional ensemble' of attention heads and MLP layers. Manipulating these components repairs over 90% of missed citations and eliminates 69% of spurious ones on PopQA without harming accuracy.
Retrieval-Augmented Generation (RAG) aims to enhance the trustworthiness of Large Language Models (LLMs) by grounding their outputs in external documents, often using inline citations for verifiability. However, the faithfulness of these citations -- whether the model genuinely uses a source to generate an answer -- remains a critical, unverified assumption. This paper offers the first mechanistic account of how a large language model decides whether to attach an inline citation while answering a factoid question. Using the Llama-3.1-8B-Instruct model in a controlled experimental environment based on the PopQA dataset, we employ an activation patching approach. We map the underlying mechanism responsible for citation, discovering that it is not a single, localized component but a distributed, multi-stage "attributional ensemble" of attention heads and MLP layers. We show that amplifying or attenuating only those critical heads and MLPs repairs over 90% of missed citations and eliminates 69% of spurious ones on PopQA without harming answer accuracy. Although gains on the multi-document HotpotQA benchmark are modest, the same component set still moves citation rates in the intended direction, indicating that the underlying mechanism is not dataset-specific. The results reveal a potential disconnect between the model's apparent reasoning and its internal computational pathway, suggesting that inline citations can create a false sense of security.