Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM
For LLM developers, this provides a method to reduce factual errors in multi-step reasoning, though it is an incremental improvement over existing RL-based approaches.
The paper identifies context-sensitive factual hallucinations in LLM reasoning traces and proposes SSC-GRPO, which assigns step-level rewards via self-consistency scores, achieving state-of-the-art performance on math reasoning benchmarks and hallucination leaderboards.
With the rapid advancement of large language models (LLMs), modern systems not only possess strong foundational capabilities and extensive knowledge, but can also solve complex problems via long, multi-step reasoning. However, as reasoning traces become longer, LLMs may produce a substantial amount of hallucinated content during the reasoning process, which is often difficult to detect. In this work, we conduct a fine-grained analysis of hallucinations arising in LLM reasoning and find that the reasoning traces are particularly prone to Context-Sensitive Factual Hallucinations: cases where the model actually has the relevant knowledge, yet makes factual errors due to contextual interference during reasoning. To address this issue, we propose Step-level Self-Consistency Group Relative Policy Optimization (SSC-GRPO), which assigns step-level rewards to reasoning traces by computing self-consistency scores of individual steps across multiple rollouts. Compared with prior methods, SSC-GRPO achieves state-of-the-art performance on both mathematical reasoning benchmarks and hallucination leaderboards. Our results offer a new perspective for detecting and mitigating hallucinations in the reasoning process of large language models.