RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
This work addresses the challenge of enhancing RAG efficiency and accuracy for LLM users by unifying ranking and generation, offering a novel approach with broad applicability across domains, though it is incremental as it builds on existing RAG and instruction-tuning methods.
The paper tackles the problem of improving retrieval-augmented generation (RAG) in large language models by proposing RankRAG, a framework that instruction-tunes a single LLM for both context ranking and answer generation, resulting in significant performance gains over strong baselines like GPT-4 and ChatQA-1.5 on multiple knowledge-intensive benchmarks, with Llama3-RankRAG outperforming Llama3-ChatQA-1.5 and GPT-4 models on nine benchmarks and showing comparable performance to GPT-4 on five biomedical RAG benchmarks without domain-specific fine-tuning.
Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG). In this work, we propose a novel instruction fine-tuning framework RankRAG, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG. In particular, the instruction-tuned LLMs work surprisingly well by adding a small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on a large amount of ranking data. For generation, we compare our model with many strong baselines, including GPT-4-0613, GPT-4-turbo-2024-0409, and ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks. Specifically, our Llama3-RankRAG significantly outperforms Llama3-ChatQA-1.5 and GPT-4 models on nine knowledge-intensive benchmarks. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization to new domains.