RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generation
This addresses the lack of specialized reward models for RAG systems, which is important for developers and researchers working on reliable AI assistants, though it appears incremental as it adapts existing datasets rather than creating fundamentally new paradigms.
The paper tackles the problem that existing reward models struggle in retrieval-augmented generation (RAG) settings by introducing RAGferee, a methodology that repurposes question-answering datasets into preference pairs prioritizing groundedness, enabling training of contextual RMs; their RAG-centric RMs achieve state-of-the-art performance with a +15.5% absolute improvement on ContextualJudgeBench.
Existing Reward Models (RMs), typically trained on general preference data, struggle in Retrieval Augmented Generation (RAG) settings, which require judging responses for faithfulness to retrieved context, relevance to the user query, appropriate refusals when context is insufficient, completeness and conciseness of information. To address the lack of publicly available RAG-centric preference datasets and specialised RMs, we introduce RAGferee, a methodology that repurposes question-answering (QA) datasets into preference pairs that prioritise groundedness over stylistic features, enabling the training of contextual RMs better suited to judging RAG responses. Using RAGferee, we curate a small preference dataset of 4K samples and fine-tune RMs ranging from 7B to 24B parameters. Our RAG-centric RMs achieve state-of-the-art performance on ContextualJudgeBench, surpassing existing 70B+ RMs trained on much larger (up to 2.4M samples) general corpora, with an absolute improvement of +15.5%.