Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning
For researchers and practitioners using LLMs for code reasoning and related tasks, this work provides a method to enhance reasoning capabilities through consistency-based reinforcement learning, though it is an incremental improvement over existing RL approaches.
CodeThinker, a consistency-driven reinforcement learning framework, improves code reasoning accuracy of LLMs by addressing sparse reward and reward hacking. It achieves state-of-the-art performance, outperforming the strongest baseline by 4.3% on Qwen2.5-Coder-7B-Instruct and yielding average gains of 5.33 and 3.11 percentage points on mathematical and code reasoning tasks.
Code reasoning refers to the task of predicting the output of a program given its source code and specific inputs. It can measure the reasoning capability of large language models (LLMs) and also benefit downstream tasks such as code generation and mathematical reasoning. Existing work has verified the effectiveness of reinforcement learning on the task. However, these methods design rewards solely based on final outputs or coarse-grained signals, and neglect the inherent consistency of the stepwise reasoning process in the task. Therefore, these methods often result in sparse reward or reward hacking, which limits the full play of enhanced learning capabilities. To alleviate these issues, we propose CodeThinker, a consistency-driven reinforcement learning framework for code reasoning. Specifically, CodeThinker has three key components: (1) a stepwise reasoning-aware model training module, which utilizes a consistency tracing paradigm as a template to synthesize training data that captures the stepwise reasoning process; (2) a dynamic beam sampling strategy, which aims to improve the quality of sampled outputs under a fixed sampling budget; and (3) a consistency reward mechanism that can effectively alleviate reward hacking. Experiments on three popular benchmarks show that CodeThinker achieves state-of-the-art performance across multiple LLMs. For instance, it outperforms the strongest baseline by 4.3% in accuracy when deployed on Qwen2.5-Coder-7B-Instruct. We also validate the effectiveness of CodeThinker on downstream tasks. Results show that, without additional training, CodeThinker obtains average accuracy gains of 5.33 and 3.11 percentage points on mathematical reasoning and code reasoning tasks covering 17 programming languages, respectively.