Evaluating Retrieval Quality in Retrieval-Augmented Generation
This addresses the problem of computationally expensive and poorly correlated evaluation for retrieval models in RAG systems, offering a more efficient and effective solution for researchers and practitioners in natural language processing.
The paper tackles the challenge of evaluating retrieval quality in retrieval-augmented generation (RAG) systems by proposing eRAG, a method that uses downstream task performance as relevance labels, achieving improvements in Kendall's τ correlation from 0.168 to 0.494 and reducing GPU memory usage by up to 50 times compared to end-to-end evaluation.
Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's $τ$ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.