Citation Failure: Definition, Analysis and Efficient Mitigation
This addresses the issue of unreliable citations for users verifying LLM-generated responses, representing an incremental improvement by building on prior work to disentangle and mitigate specific failure modes.
The paper tackles the problem of citation failure in LLM-based RAG systems, where models generate helpful responses but fail to cite complete evidence, and proposes CITENTION, a framework that integrates multiple methods to achieve substantial citation improvements on the introduced CITECONTROL benchmark and in transfer settings.
Citations from LLM-based RAG systems are supposed to simplify response verification. However, this does not hold for citation failure, when a model generates a helpful response, but fails to cite complete evidence. In contrast to previous work, we propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible. To address citation failure, this work follows a two-step approach: (1) We study when citation failure occurs and (2) how it can be mitigated. For step 1, we extend prior work by investigating how the relation between response and evidence affects citation quality. We introduce CITECONTROL, a benchmark that systematically varies this relation to analyze failure modes. Experiments show that failures increase with relational complexity and suggest that combining citation methods could improve performance, motivating step 2. To improve LLM citation efficiently, we propose CITENTION, a framework integrating generative, attention-based, and retrieval-based methods. Results demonstrate substantial citation improvements on CITECONTROL and in transfer settings. We make our data and code publicly available.