SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generation
This addresses a critical challenge in RAG systems for users relying on accurate, context-faithful AI responses, representing a novel method rather than an incremental improvement.
The paper tackles the problem of faithfulness hallucination in Retrieval-Augmented Generation (RAG) systems by introducing Self-Supervised Faithfulness Optimization (SSFO), a self-supervised alignment approach that significantly outperforms existing methods and achieves state-of-the-art faithfulness on multiple context-based question-answering datasets.
Retrieval-Augmented Generation (RAG) systems require Large Language Models (LLMs) to generate responses that are faithful to the retrieved context. However, faithfulness hallucination remains a critical challenge, as existing methods often require costly supervision and post-training or significant inference burdens. To overcome these limitations, we introduce Self-Supervised Faithfulness Optimization (SSFO), the first self-supervised alignment approach for enhancing RAG faithfulness. SSFO constructs preference data pairs by contrasting the model's outputs generated with and without the context. Leveraging Direct Preference Optimization (DPO), SSFO aligns model faithfulness without incurring labeling costs or additional inference burden. We theoretically and empirically demonstrate that SSFO leverages a benign form of \emph{likelihood displacement}, transferring probability mass from parametric-based tokens to context-aligned tokens. Based on this insight, we propose a modified DPO loss function to encourage likelihood displacement. Comprehensive evaluations show that SSFO significantly outperforms existing methods, achieving state-of-the-art faithfulness on multiple context-based question-answering datasets. Notably, SSFO exhibits strong generalization, improving cross-lingual faithfulness and preserving general instruction-following capabilities. We release our code and model at the anonymous link: https://github.com/chkwy/SSFO