CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation
This addresses a specific bottleneck in RAG systems for users relying on LLMs for grounded generation, but it is incremental as it builds on prior work on position perturbation.
The paper tackles the problem of position bias in retrieval-augmented generation (RAG) for large language models, which causes uneven attention to retrieved contexts, and proposes CORD, a method that balances consistency regularization and rank distillation to improve robustness, achieving outperformance in diverse RAG benchmarks.
With the adoption of retrieval-augmented generation (RAG), large language models (LLMs) are expected to ground their generation to the retrieved contexts. Yet, this is hindered by position bias of LLMs, failing to evenly attend to all contexts. Previous work has addressed this by synthesizing contexts with perturbed positions of gold segment, creating a position-diversified train set. We extend this intuition to propose consistency regularization with augmentation and distillation. First, we augment each training instance with its position perturbation to encourage consistent predictions, regardless of ordering. We also distill behaviors of this pair, although it can be counterproductive in certain RAG scenarios where the given order from the retriever is crucial for generation quality. We thus propose CORD, balancing COnsistency and Rank Distillation. CORD adaptively samples noise-controlled perturbations from an interpolation space, ensuring both consistency and respect for the rank prior. Empirical results show this balance enables CORD to outperform consistently in diverse RAG benchmarks.