Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
This addresses the issue of insufficient exploration capability in RLVR for LLMs, offering a domain-specific improvement.
The paper tackles the problem of limited exploration diversity in Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models by proposing MENTOR, a framework that provides expert guidance only at critical decision points, resulting in superior overall performance.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.