Coupled Variational Reinforcement Learning for Language Model General Reasoning
This work addresses the problem of inefficient exploration and thought-answer incoherence in language model reasoning for AI researchers, offering an incremental improvement over existing verifier-free RL methods.
The paper tackles the inefficiency and incoherence in verifier-free reinforcement learning for language model reasoning by proposing CoVRL, which couples prior and posterior distributions through a hybrid sampling strategy, resulting in a 12.4% improvement over the base model and a 2.3% gain over state-of-the-art baselines.
While reinforcement learning has achieved impressive progress in language model reasoning, it is constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the probabilities that LLMs generate reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \textit{\b{Co}upled \b{V}ariational \b{R}einforcement \b{L}earning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.