Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
This addresses the problem of evaluating hybrid RAG systems for researchers and practitioners, but it is incremental as a survey paper.
This paper tackles the challenge of evaluating Retrieval-Augmented Generation (RAG) systems in the era of Large Language Models by providing a comprehensive survey of evaluation methods, datasets, and frameworks, resulting in a critical resource for advancing RAG development.
Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual accuracy, safety, and computational efficiency in the LLM era. We also compile and categorize the RAG-specific datasets and evaluation frameworks, conducting a meta-analysis of evaluation practices in high-impact RAG research. To the best of our knowledge, this work represents the most comprehensive survey for RAG evaluation, bridging traditional and LLM-driven methods, and serves as a critical resource for advancing RAG development.