Learning to Reason Faithfully through Step-Level Faithfulness Maximization
This work addresses the issue of unreliable intermediate reasoning steps in AI systems, which is crucial for improving trust and accuracy in applications like question-answering and decision-making, though it is incremental as it builds upon existing RLVR methods.
The paper tackled the problem of spurious reasoning and hallucinations in Large Language Models during multi-step tasks by proposing FaithRL, a reinforcement learning framework that optimizes reasoning faithfulness, resulting in reduced hallucination rates while maintaining or improving answer correctness across diverse benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) has markedly improved the performance of Large Language Models (LLMs) on tasks requiring multi-step reasoning. However, most RLVR pipelines rely on sparse outcome-based rewards, providing little supervision over intermediate steps and thus encouraging over-confidence and spurious reasoning, which in turn increases hallucinations. To address this, we propose FaithRL, a general reinforcement learning framework that directly optimizes reasoning faithfulness. We formalize a faithfulness-maximization objective and theoretically show that optimizing it mitigates over-confidence. To instantiate this objective, we introduce a geometric reward design and a faithfulness-aware advantage modulation mechanism that assigns step-level credit by penalizing unsupported steps while preserving valid partial derivations. Across diverse backbones and benchmarks, FaithRL consistently reduces hallucination rates while maintaining (and often improving) answer correctness. Further analysis confirms that FaithRL increases step-wise reasoning faithfulness and generalizes robustly. Our code is available at https://github.com/aintdoin/FaithRL.