Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment
This work tackles the critical problem of effective reasoning with long contexts for RAG models, which is a significant challenge for researchers and practitioners deploying these systems.
This paper addresses attention dilution and reasoning hallucinations in Retrieval-Augmented Generation (RAG) when dealing with extensive contexts. The proposed Hit-RAG framework, a multi-stage preference alignment pipeline, significantly improves performance across eight benchmarks, allowing models to reason accurately with long contexts and outperform larger models.
Despite the promise of Retrieval-Augmented Generation in grounding Multimodal Large Language Models with external knowledge, the transition to extensive contexts often leads to significant attention dilution and reasoning hallucinations. The surge in information density causes critical evidence to be submerged by voluminous noise, which complicates the discernment of relevant fragments within a dense input. In this paper, we propose \textbf{Hit-RAG}, a multi-stage preference alignment framework designed to resolve these cognitive bottlenecks through a progressive optimization pipeline. Our approach systematically refines the utilization of external evidence via three distinct stages. First, Supervised Fine-tuning establishes baseline context awareness to minimize information neglect. Next, Discriminative Preference Alignment enhances robustness against misleading distractors. Finally, Group-Relative Policy Optimization stabilizes logical synthesis to prevent reasoning collapse. Extensive evaluations on eight benchmarks demonstrate that Hit-RAG consistently yields substantial performance gains, enabling models to bridge the gap between context acquisition and accurate reasoning while surpassing much larger counterparts in long-context scenarios.