RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation
This work addresses the problem of deploying RAG in real-world, diverse retrieval scenarios for AI practitioners, revealing critical limitations that are incremental to existing benchmark-focused research.
The study evaluated retrieval-augmented generation (RAG) systems using a large-scale datastore with mixed knowledge, finding that retrieval mainly benefits smaller models, rerankers add minimal value, and no single source excels consistently, with LLMs struggling to route queries across heterogeneous sources.
Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge retrieved at inference time. While RAG demonstrates strong performance on benchmarks largely derived from general-domain corpora like Wikipedia, its effectiveness under realistic, diverse retrieval scenarios remains underexplored. We evaluated RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge, and identified critical limitations: retrieval mainly benefits smaller models, rerankers add minimal value, and no single retrieval source consistently excels. Moreover, current LLMs struggle to route queries across heterogeneous knowledge sources. These findings highlight the need for adaptive retrieval strategies before deploying RAG in real-world settings. Our code and data can be found at https://github.com/ritaranx/RAG_in_the_Wild.