Long Context vs. RAG for LLMs: An Evaluation and Revisits
This work provides guidance for optimizing LLMs with external knowledge sources by highlighting trade-offs between LC and RAG strategies, though it is incremental as it revisits and expands on prior studies.
This paper evaluates and compares two strategies for enabling LLMs to use long external contexts—Long Context (LC) and Retrieval-Augmented Generation (RAG)—finding that LC generally outperforms RAG in question-answering benchmarks, especially for Wikipedia-based questions, while RAG has advantages in dialogue-based and general queries.
Extending context windows (i.e., Long Context, LC) and using retrievers to selectively access relevant information (i.e., Retrieval-Augmented Generation, RAG) are the two main strategies to enable LLMs to incorporate extremely long external contexts. This paper revisits recent studies on this topic, highlighting their key insights and discrepancies. We then provide a more comprehensive evaluation by filtering out questions answerable without external context, identifying the most effective retrieval methods, and expanding the datasets. We show that LC generally outperforms RAG in question-answering benchmarks, especially for Wikipedia-based questions. Summarization-based retrieval performs comparably to LC, while chunk-based retrieval lags behind. However, RAG has advantages in dialogue-based and general question queries. These insights underscore the trade-offs between RAG and LC strategies, offering guidance for future optimization of LLMs with external knowledge sources. We also provide an in-depth discussion on this topic, highlighting the overlooked importance of context relevance in existing studies.