Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
This work addresses the problem of limited exploration and performance bottlenecks in reinforcement learning for multi-domain reasoning in large language models, representing a novel method rather than an incremental improvement.
The paper tackles the limitations of existing reinforcement learning methods for large language models, which focus on single-domain reasoning with verifiable rewards and restrict exploration, by proposing RGR-GRPO, a rubric-driven framework that provides fine-grained rewards and offline guidance for multi-domain reasoning, resulting in average improvements of +5.4% to +8.4% across mathematics, physics, chemistry, and general reasoning tasks.
Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose $\textbf{RGR-GRPO}$ (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.