Prompt Curriculum Learning for Efficient LLM Post-Training
This addresses efficiency bottlenecks in RL-based post-training for language models, offering a lightweight method to reduce computational costs, though it is incremental as it builds on existing RL and curriculum learning approaches.
The paper tackles the problem of inefficient prompt selection in post-training large language models via reinforcement learning by introducing Prompt Curriculum Learning (PCL), which uses a learned value model to select intermediate-difficulty prompts, achieving up to 16.9x faster speed in identifying such prompts while maintaining or improving performance.
We introduce Prompt Curriculum Learning (PCL), a lightweight reinforcement learning (RL) algorithm that selects intermediate-difficulty prompts using a learned value model to post-train language models. Since post-training LLMs via RL remains sensitive to batching and prompt selection strategies, we first conduct a series of systematic experiments where we (1) determine the optimal training batch size that balances generation efficiency and gradient quality and (2) establish the importance of focusing on prompts of intermediate difficulty for the policy. We build upon these results to design PCL, which identifies prompts of intermediate difficulty for the current policy in an on-policy manner by using a value model that is concurrently updated based on the current policy. By focusing on informative prompts that yield high effective ratios, PCL achieves either the highest performance or requires significantly less time to reach comparable performance to its counterparts. Compared to rollout-based filtering methods, PCL avoids costly rollouts and achieves $12.1\times$ and $16.9\times$ faster speed on identifying intermediate-difficulty prompts when training on MATH and DeepScaleR, respectively. We further demonstrate that our value model accurately predicts prompt difficulty and allows PCL to focus on progressively more challenging prompts during RL. Our results present a new methodology that delivers improved tradeoff between upper-bound performance and efficiency for reasoning-focused RL.