From Facts to Conclusions : Integrating Deductive Reasoning in Retrieval-Augmented LLMs
This addresses the issue of unreliable information in RAG systems for users needing accurate and interpretable AI-generated answers, though it is incremental as it builds on prior work with a unified approach.
The paper tackled the problem of Retrieval-Augmented Generation (RAG) failing with conflicting or unreliable sources by proposing a reasoning-trace-augmented framework, resulting in substantial gains such as improving answer correctness from 0.069 to 0.883 and behavioral adherence from 0.074 to 0.722 in experiments.
Retrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external evidence, but fails when retrieved sources conflict or contain outdated or subjective information. Prior work address these issues independently but lack unified reasoning supervision. We propose a reasoning-trace-augmented RAG framework that adds structured, interpretable reasoning across three stages : (1) document-level adjudication, (2) conflict analysis, and (3) grounded synthesis, producing citation-linked answers or justified refusals. A Conflict-Aware Trust-Score (CATS) pipeline is introduced which evaluates groundedness, factual correctness, refusal accuracy, and conflict-behavior alignment using an LLM-as-a-Judge. Our 539-query reasoning dataset and evaluation pipeline establish a foundation for conflict-aware, interpretable RAG systems. Experimental results demonstrate substantial gains over baselines, most notably with Qwen, where Supervised Fine-Tuning improved End-to-End answer correctness from 0.069 to 0.883 and behavioral adherence from 0.074 to 0.722.