SFR-RAG: Towards Contextually Faithful LLMs
This addresses the challenge of contextually faithful generation in RAG for AI applications, offering a more efficient and reliable model, though it is incremental as it builds on existing RAG paradigms.
The paper tackles the problem of improving factual accuracy and reducing hallucinations in Retrieval Augmented Generation (RAG) systems by introducing SFR-RAG, a small instruction-tuned LLM, which outperforms larger models like Command-R+ (104B) and GPT-4o, achieving state-of-the-art results in 3 out of 7 benchmarks with significantly fewer parameters.
Retrieval Augmented Generation (RAG), a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance, has emerged as a pivotal area in generative AI. The LLMs used in RAG applications are required to faithfully and completely comprehend the provided context and users' questions, avoid hallucination, handle unanswerable, counterfactual or otherwise low-quality and irrelevant contexts, perform complex multi-hop reasoning and produce reliable citations. In this paper, we introduce SFR-RAG, a small LLM that is instruction-tuned with an emphasis on context-grounded generation and hallucination minimization. We also present ContextualBench, a new evaluation framework compiling multiple popular and diverse RAG benchmarks, such as HotpotQA and TriviaQA, with consistent RAG settings to ensure reproducibility and consistency in model assessments. Experimental results demonstrate that our SFR-RAG-9B model outperforms leading baselines such as Command-R+ (104B) and GPT-4o, achieving state-of-the-art results in 3 out of 7 benchmarks in ContextualBench with significantly fewer parameters. The model is also shown to be resilient to alteration in the contextual information and behave appropriately when relevant context is removed. Additionally, the SFR-RAG model maintains competitive performance in general instruction-following tasks and function-calling capabilities.