From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
This addresses the issue of unreliable self-verification in LLMs for reasoning tasks, but it is incremental as it builds on existing RL methods.
The paper tackles the problem of LLMs struggling to consistently verify their own reasoning traces by proposing GRPO-Verif, an algorithm that jointly optimizes solution generation and self-verification, resulting in enhanced self-verification capability while maintaining comparable reasoning performance.
The reasoning capabilities of large language models (LLMs) have been significantly improved through reinforcement learning (RL). Nevertheless, LLMs still struggle to consistently verify their own reasoning traces. This raises the research question of how to enhance the self-verification ability of LLMs and whether such an ability can further improve reasoning performance. In this work, we propose GRPO-Verif, an algorithm that jointly optimizes solution generation and self-verification within a unified loss function, with an adjustable hyperparameter controlling the weight of the verification signal. Experimental results demonstrate that our method enhances self-verification capability while maintaining comparable performance in reasoning.