Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering
This work addresses the need for reliable and accurate medical QA systems by providing a domain-specific evaluation framework, though it is incremental as it builds on existing RAG benchmarks.
The paper tackles the lack of comprehensive evaluation for retrieval-augmented generation (RAG) systems in medical question answering by introducing MedRGB, a benchmark that tests models on practical scenarios like sufficiency, integration, and robustness, revealing their limited ability to handle noise and misinformation in retrieved documents.
Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) in knowledge-intensive tasks such as those from medical domain. However, the sensitive nature of the medical domain necessitates a completely accurate and trustworthy system. While existing RAG benchmarks primarily focus on the standard retrieve-answer setting, they overlook many practical scenarios that measure crucial aspects of a reliable medical system. This paper addresses this gap by providing a comprehensive evaluation framework for medical question-answering (QA) systems in a RAG setting for these situations, including sufficiency, integration, and robustness. We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets for testing LLMs' ability to handle these specific scenarios. Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art commercial LLMs and open-source models across multiple retrieval conditions. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents. We further analyze the LLMs' reasoning processes to provides valuable insights and future directions for developing RAG systems in this critical medical domain.