RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering
This work addresses the problem of ensuring clinical accuracy and patient safety in AI applications for healthcare by providing a systematic diagnosis of retrieval-augmented generation, which is an incremental improvement over existing evaluation methods.
The authors tackled the problem of evaluating retrieval-augmented generation for medical question answering and found a 14% gap between perceived system success and evidence-based grounding. Their proposed framework, RAG-X, provides diagnostic transparency for safe and verifiable clinical RAG systems.
Automated question-answering (QA) systems increasingly rely on retrieval-augmented generation (RAG) to ground large language models (LLMs) in authoritative medical knowledge, ensuring clinical accuracy and patient safety in Artificial Intelligence (AI) applications for healthcare. Despite progress in RAG evaluation, current benchmarks focus only on simple multiple-choice QA tasks and employ metrics that poorly capture the semantic precision required for complex QA tasks. These approaches fail to diagnose whether an error stems from faulty retrieval or flawed generation, limiting developers from performing targeted improvement. To address this gap, we propose RAG-X, a diagnostic framework that evaluates the retriever and generator independently across a triad of QA tasks: information extraction, short-answer generation, and multiple-choice question (MCQ) answering. RAG-X introduces Context Utilization Efficiency (CUE) metrics to disaggregate system success into interpretable quadrants, isolating verified grounding from deceptive accuracy. Our experiments reveal an ``Accuracy Fallacy", where a 14\% gap separates perceived system success from evidence-based grounding. By surfacing hidden failure modes, RAG-X offers the diagnostic transparency needed for safe and verifiable clinical RAG systems.