LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
For practitioners of RL post-training, the paper provides actionable design rules and a method to automatically discover adaptive strategies, though the approach is incremental over existing hyperparameter optimization methods.
The paper identifies a structural principle in RL post-training: capacity parameters increase monotonically while regularization parameters oscillate. LLMZero, an LLM-agent system using tree search, discovers adaptive training strategies that outperform baselines by 9-140% relative across 4 GRPO tasks.
RL post-training strategies are dataset-dependent and reveal a recurring empirical pattern: capacity parameters accumulate monotonically across stages, while regularization parameters predominantly oscillate in response to shifting training dynamics. This distinction matters because fixed schedules commit all parameters to fixed trajectories and therefore cannot express the non-stationary exploration-exploitation tradeoffs that regularization must track; the principle provides actionable design rules for multi-stage training. We discover this through LLMZero, a system where LLM agents search over training trajectories via tree search, diagnosing pathologies at each checkpoint and proposing coordinated multi-parameter transitions. Across 4 diverse GRPO tasks, LLMZero discovers strategies that improve over the base model by 9% to 140% relative and over grid search by 6% to 15% relative, consistently outperforming random search and the skill-based agent. The structural principle transfers across tasks, providing an explanation for why discovered strategies take qualitatively different forms yet share similar parameter dynamics.