Escaping the Verifier: Learning to Reason via Demonstrations
This addresses the challenge of enabling robust reasoning learning in LLMs for real-world tasks where verifiers are unavailable, though it is incremental as it builds on existing RL and demonstration-based approaches.
The paper tackles the problem of training Large Language Models (LLMs) for reasoning tasks without task-specific verifiers by introducing RARO, a method that learns from expert demonstrations via Inverse Reinforcement Learning, achieving significant performance improvements over verifier-free baselines on tasks like Countdown, DeepMath, and Poetry Writing.
Training Large Language Models (LLMs) to reason often relies on Reinforcement Learning (RL) with task-specific verifiers. However, many real-world reasoning-intensive tasks lack verifiers, despite offering abundant expert demonstrations that remain under-utilized for reasoning-focused training. We introduce RARO (Relativistic Adversarial Reasoning Optimization) that learns strong reasoning capabilities from only expert demonstrations via Inverse Reinforcement Learning. Our method sets up an adversarial game between a policy and a relativistic critic: the policy learns to mimic expert answers, while the critic aims to identify the experts among (expert, policy) answer pairs. Both the policy and the critic are trained jointly and continuously via RL, and we identify the key stabilization techniques required for robust learning. Empirically, RARO significantly outperforms strong verifier-free baselines on all of our evaluation tasks -- Countdown, DeepMath, and Poetry Writing -- and enjoys the same robust scaling trends as RL with verifiers. These results demonstrate that our method effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable.