MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation
This addresses the problem of limited benchmarks for detailed RAG evaluation for researchers, though it is incremental as it builds on existing RAG frameworks.
The authors tackled the challenge of evaluating Retrieval-Augmented Generation (RAG) systems by introducing MIRAGE, a dataset with 7,560 instances and a retrieval pool of 37,800 entries, along with novel metrics to assess adaptability, providing insights into optimal model pairings.
Retrieval-Augmented Generation (RAG) has gained prominence as an effective method for enhancing the generative capabilities of Large Language Models (LLMs) through the incorporation of external knowledge. However, the evaluation of RAG systems remains a challenge, due to the intricate interplay between retrieval and generation components. This limitation has resulted in a scarcity of benchmarks that facilitate a detailed, component-specific assessment. In this work, we present MIRAGE, a Question Answering dataset specifically designed for RAG evaluation. MIRAGE consists of 7,560 curated instances mapped to a retrieval pool of 37,800 entries, enabling an efficient and precise evaluation of both retrieval and generation tasks. We also introduce novel evaluation metrics aimed at measuring RAG adaptability, encompassing dimensions such as noise vulnerability, context acceptability, context insensitivity, and context misinterpretation. Through comprehensive experiments across various retriever-LLM configurations, we provide new insights into the optimal alignment of model pairs and the nuanced dynamics within RAG systems. The dataset and evaluation code are publicly available, allowing for seamless integration and customization in diverse research settings\footnote{The MIRAGE code and data are available at https://github.com/nlpai-lab/MIRAGE.