MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine
This provides a domain-specific benchmark for evaluating RAG systems in biomedicine, which is incremental as it adapts existing RAG concepts to a new application area.
The authors tackled the lack of a comprehensive evaluation benchmark for Retrieval-Augmented Generation (RAG) in the medical domain by introducing the MRAG benchmark, which includes tasks in English and Chinese and shows that RAG enhances LLM reliability across these tasks.
While Retrieval-Augmented Generation (RAG) has been swiftly adopted in scientific and clinical QA systems, a comprehensive evaluation benchmark in the medical domain is lacking. To address this gap, we introduce the Medical Retrieval-Augmented Generation (MRAG) benchmark, covering various tasks in English and Chinese languages, and building a corpus with Wikipedia and Pubmed. Additionally, we develop the MRAG-Toolkit, facilitating systematic exploration of different RAG components. Our experiments reveal that: (a) RAG enhances LLM reliability across MRAG tasks. (b) the performance of RAG systems is influenced by retrieval approaches, model sizes, and prompting strategies. (c) While RAG improves usefulness and reasoning quality, LLM responses may become slightly less readable for long-form questions. We will release the MRAG-Bench's dataset and toolkit with CCBY-4.0 license upon acceptance, to facilitate applications from both academia and industry.