Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation
This addresses the issue of noisy retrieval impacting generation quality in RAG systems for LLM applications, representing an incremental improvement over previous denoising methods.
The paper tackles the problem of retrieval noise in Retrieval-Augmented Generation (RAG) by proposing EviOmni, a method that learns to extract rational evidence through explicit reasoning and conscious extraction, which improves accuracy on downstream tasks in experiments across three benchmark datasets.
Retrieval-Augmented Generation (RAG) effectively improves the accuracy of Large Language Models (LLMs). However, retrieval noises significantly impact the quality of LLMs' generation, necessitating the development of denoising mechanisms. Previous methods extract evidence straightforwardly without explicit thinking, which risks filtering out key clues and struggles with generalization. To this end, we propose EviOmni, which learns to extract rational evidence by (1) explicitly reasoning to identify potential cues within retrieval contents first, and then (2) consciously extracting to avoid omitting any key cues helpful for answering questions. Specifically, we frame evidence reasoning and evidence extraction into one unified response for end-to-end training; apply knowledge token masks for disentanglement to derive reasoning-based and extraction-based answers; and devise three types of verifiable reward functions, including answer, length, and format, to update the model via the policy optimization algorithm. Extensive experiments on three benchmark datasets show the effectiveness of EviOmni, providing compact and high-quality evidence, improving the accuracy of downstream tasks, and promoting effective application in online RAG systems.