Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
This work addresses the problem of inefficient resource usage in GRPO training for large reasoning models, offering a practical guide for researchers and practitioners, though it is incremental as it builds on existing scaling law concepts.
The paper tackles the high computational cost of fine-tuning large language models for reasoning tasks using GRPO by proposing a predictive framework that models training dynamics, deriving an empirical scaling law that predicts reward trajectories and identifies three training phases, and finding that early stopping can reduce compute by up to 30% without performance loss.
Fine-tuning large language models (LLMs) for reasoning tasks using reinforcement learning methods like Group Relative Policy Optimization (GRPO) is computationally expensive. To address this, we propose a predictive framework that models training dynamics and helps optimize resource usage. Through experiments on Llama and Qwen models (3B 8B), we derive an empirical scaling law based on model size, initial performance, and training progress. This law predicts reward trajectories and identifies three consistent training phases: slow start, rapid improvement, and plateau. We find that training beyond certain number of an epoch offers little gain, suggesting earlier stopping can significantly reduce compute without sacrificing performance. Our approach generalizes across model types, providing a practical guide for efficient GRPO-based fine-tuning.