MSRS: Evaluating Multi-Source Retrieval-Augmented Generation
This addresses the need for better evaluation of RAG systems in real-world applications requiring multi-source integration, though it is incremental as it focuses on benchmarking rather than novel method development.
The paper tackles the problem of evaluating retrieval-augmented generation (RAG) systems in multi-source settings where information is scattered across sources, by introducing a scalable framework to build benchmarks (MSRS-Story and MSRS-Meet) and showing that generation quality heavily depends on retrieval effectiveness, with reasoning models outperforming standard LLMs in synthesis.
Retrieval-augmented systems are typically evaluated in settings where information required to answer the query can be found within a single source or the answer is short-form or factoid-based. However, many real-world applications demand the ability to integrate and summarize information scattered across multiple sources, where no single source is sufficient to respond to the user's question. In such settings, the retrieval component of a RAG pipeline must recognize a variety of relevance signals, and the generation component must connect and synthesize information across multiple sources. We present a scalable framework for constructing evaluation benchmarks that challenge RAG systems to integrate information across distinct sources and generate long-form responses. Using our framework, we build two new benchmarks on Multi-Source Retrieval and Synthesis: MSRS-Story and MSRS-Meet, representing narrative synthesis and summarization tasks, respectively, that require retrieval from large collections. Our extensive experiments with various RAG pipelines -- including sparse and dense retrievers combined with frontier LLMs -- reveal that generation quality is highly dependent on retrieval effectiveness, which varies greatly by task. While multi-source synthesis proves challenging even in an oracle retrieval setting, we find that reasoning models significantly outperform standard LLMs at this distinct step.