Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets
This provides practical guidance for budget-constrained fine-tuning of language models, though it is incremental as it focuses on optimizing existing GRPO methods rather than introducing new paradigms.
The paper tackled the problem of maximizing GRPO post-training effectiveness under limited annotation budgets by investigating how example difficulty affects learning, finding that training on the hardest 10% of examples yields performance gains up to 47% while easy examples produce minimal improvements of 3-15%.
Collecting high-quality training examples for language model fine-tuning is expensive, with practical budgets limiting the amount of data that can be procured. We investigate whether example difficulty affects GRPO training effectiveness by comparing selection strategies (easy, medium, hard, random) across multiple models and reasoning tasks. Training on the hardest 10\% of examples (those where the base model fails most often) yields dramatic performance gains up to 47\%, while easy examples produce minimal improvements of 3-15\%. This occurs because GRPO requires outcome variance to generate learning signals; hard examples maintain mixed success/failure outcomes throughout training while easy examples quickly converge to consistent success, eliminating learning opportunities. Moreover, models trained on hard examples show superior out-of-distribution generalization, with only hard-trained models achieving meaningful gains on the AIME2025 benchmark. Our findings provide clear guidance: when budget-constrained, prioritize collecting and annotating examples where your base model struggles, as these drive nearly all learning value in GRPO fine-tuning