Context-DPO: Aligning Language Models for Context-Faithfulness
This addresses the issue of unreliable responses in LLMs for users relying on retrieval-augmented generation, though it is incremental as it builds on existing alignment techniques.
The paper tackles the problem of improving context-faithfulness in large language models (LLMs) by proposing Context-DPO, an alignment method that enhances adherence to retrieved information, achieving 35% to 280% improvements on open-source models.
Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at https://github.com/byronBBL/Context-DPO