OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning
For LLM reasoning tasks, OGER addresses the exploration bottleneck by unifying offline and online RL, offering a practical method to improve performance without requiring new data.
OGER introduces a hybrid reinforcement learning framework that combines offline teacher guidance with online exploration via an entropy-aware reward, achieving significant gains in mathematical reasoning and robust generalization on out-of-domain tasks.
Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have significantly improved Large Language Model (LLM) reasoning, yet models often struggle to explore novel trajectories beyond their initial latent space. While offline teacher guidance and entropy-driven strategies have been proposed to address this, they often lack deep integration or are constrained by the model's inherent capacity. In this paper, we propose OGER, a novel framework that unifies offline teacher guidance and online reinforcement learning through a specialized reward modeling lens. OGER employs multi-teacher collaborative training and constructs an auxiliary exploration reward that leverages both offline trajectories and the model's own entropy to incentivize autonomous exploration. Extensive experiments across mathematical and general reasoning benchmarks demonstrate that OGER significantly outperforms competitive baselines, achieving substantial gains in mathematical reasoning while maintaining robust generalization to out-of-domain tasks. We provide a comprehensive analysis of training dynamics and conduct detailed ablation studies to validate the effectiveness of our entropy-aware reward modulation. Our code is available at https://github.com/ecoli-hit/OGER.git.