Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
This work provides a new method for improving the efficiency and performance of LLM training for practitioners using GRPO, by leveraging smaller models for exploration.
This paper addresses the problem of enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs, proposing that smaller models naturally exhibit higher policy-level diversity. They introduce S2L-PO, a framework that leverages smaller models as explorers to train larger models, achieving an 8.8% accuracy improvement on AIME 24 using a 1.7B explorer for an 8B model.
We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and lead to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner's own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (e.g., +8.8% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute.