Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning
For researchers training LLMs on reasoning tasks, PieceHint offers a method to improve learning efficiency and performance without sacrificing output diversity.
PieceHint addresses the dilemma in reinforcement learning for LLMs where training on easy problems causes overfitting and hard problems yield sparse rewards. By strategically injecting hints at critical reasoning steps and progressively withdrawing them, a 1.5B model achieves comparable average performance to 32B baselines on six math benchmarks while preserving pass@k diversity.
Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training on hard problems often results in sparse rewards. Recent question augmentation methods address this by prepending partial solutions as hints. However, uniform hint provision may introduce redundant information while missing critical reasoning bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. We propose \textbf{PieceHint}, a hint injection framework that strategically identifies and provides critical reasoning steps during training. By scoring the importance of different reasoning steps, selectively allocating hints based on problem difficulty, and progressively withdrawing scaffolding, PieceHint enables models to transition from guided learning to independent reasoning. Experiments on six mathematical reasoning benchmarks show that our 1.5B model achieves comparable average performance to 32B baselines while preserving pass@k diversity across all $k$ values.