Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics
This addresses a key bottleneck in reinforcement learning for LLM reasoning by reducing reliance on human labels, though it is incremental as it builds on existing self-evolving training methods.
The paper tackles the problem of autonomously rewarding chain-of-thought reasoning in LLMs without human annotation, proposing RLCER which uses self-evolving rubrics to outperform outcome-centric methods and improve inference-time performance.
Despite chain-of-thought (CoT) playing crucial roles in LLM reasoning, directly rewarding it is difficult: training a reward model demands heavy human labeling efforts, and static RMs struggle with evolving CoT distributions and reward hacking. These challenges motivate us to seek an autonomous CoT rewarding approach that requires no human annotation efforts and can evolve gradually. Inspired by recent self-evolving training methods, we propose \textbf{RLCER} (\textbf{R}einforcement \textbf{L}earning with \textbf{C}oT Supervision via Self-\textbf{E}volving \textbf{R}ubrics), which enhances the outcome-centric RLVR by rewarding CoTs with self-proposed and self-evolving rubrics. We show that self-proposed and self-evolving rubrics provide reliable CoT supervision signals even without outcome rewards, enabling RLCER to outperform outcome-centric RLVR. Moreover, when used as in-prompt hints, these self-proposed rubrics further improve inference-time performance.