Teaching Large Reasoning Models Effective Reflection
This work addresses a key inefficiency in reasoning models for AI applications, offering a method to make self-reflection more effective, though it is incremental as it builds on existing self-critique techniques.
The paper tackles the problem of superficial reflection in Large Reasoning Models, which often leads to wasted computation without improving answers, and introduces Self-Critique Fine-Tuning (SCFT) and Reinforcement Learning with Effective Reflection Rewards (RLERR) to enhance reflective reasoning, resulting in significant improvements in reasoning accuracy and reflection quality on benchmarks like AIME2024 and AIME2025, outperforming state-of-the-art baselines.
Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks, often by engaging in self-reflective behaviors such as self-critique and backtracking. However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer and incurring computation overhead. In this paper, we identify and address the problem of superficial reflection in LRMs. We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques. SCFT prompts models to critique their own outputs, filters high-quality critiques through rejection sampling, and fine-tunes the model using a critique-based objective. Building on this strong foundation, we further introduce Reinforcement Learning with Effective Reflection Rewards (RLERR). RLERR leverages the high-quality reflections initialized by SCFT to construct reward signals, guiding the model to internalize the self-correction process via reinforcement learning. Experiments on two challenging benchmarks, AIME2024 and AIME2025, show that SCFT and RLERR significantly improve both reasoning accuracy and reflection quality, outperforming state-of-the-art baselines. All data and codes are available at https://github.com/wanghanbinpanda/SCFT.