Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG
For developers and deployers of multi-source RAG systems, this work provides a method to audit source-dependence, a critical but overlooked evaluation axis, though the contribution is incremental as it extends existing evaluation paradigms.
The paper identifies source-dependence as a failure mode in multi-source RAG systems, where different sources yield different answers to the same question, and proposes a framework to audit this. Using a benchmark of patient questions from transplant handbooks, they show that better retrieval reveals more disagreement than previously estimated, with the framework transferring to legal and educational domains.
A retrieval-augmented generation (RAG) system deployed over a multi-author institutional corpus can give a different answer to the same question depending on which source it retrieves -- a failure mode the dominant single-gold-answer paradigm cannot diagnose. We argue that source-dependence is a missing axis of NLP evaluation, and that auditing it means shifting the unit of evaluation from answer correctness to the inter-source relationship. We make this concrete in transplant patient education, where institutional sources demonstrably disagree, releasing three artefacts: TransplantQA, a benchmark of real patient questions, each answered by grounding generation in multiple institutional handbooks as candidate sources; HERO-QA, a hierarchical retrieval strategy that grounds and audits each answer; and a structured-output judge that scores inter-source relationships on a validated 5-label taxonomy. At scale, better retrieval reveals far more disagreement than prior estimates suggested -- understating its prevalence, not its intensity. The framework is domain-agnostic and transfers to legal and educational RAG: measuring source-dependence is a responsibility for deployed multi-source NLP generally.