Learning What to Learn: Stage-Specific Data Sets for SFT-then-RL in Small Language Model Reasoning
For researchers working on post-training small language models for reasoning, this work provides a principled data strategy that aligns data difficulty with the distinct roles of SFT and RL, leading to improved performance.
The paper proposes a difficulty-aware SFT-then-RL framework for post-training small language models on reasoning tasks, where SFT focuses on acquiring new skills and RL consolidates existing ones. Experiments on two SLMs across five benchmarks show consistent improvements over representative baselines.
Post-training Small Language Models (SLMs) for reasoning typically follows an SFT-then-RL pipeline, yet existing work rarely considers what data should be learned at each stage. We argue that data strategy should be aligned with the distinct roles of SFT and RL: SFT is better suited for acquiring not-yet-mastered reasoning skills, while RL is better suited for consolidating skills that the model can already partially access. Based on this principle, we propose a difficulty-aware SFT-then-RL framework that organizes training data into stage-specific sets. For hard samples in the SFT stage, we introduce a Bridge mechanism that transforms raw teacher-generated reasoning traces into more learnable supervision for SLMs. For hard samples that remain unsolved during RL, we apply Critique Fine-Tuning by converting all-zero-reward failures into diagnostic, repair, and new reasoning trace supervision for the next SFT stage. Experiments on two SLMs across five reasoning benchmarks show that our method consistently improves over representative SFT, distillation, and RL baselines. Our results highlight the importance of coordinating data difficulty across SFT and RL for effective SLM reasoning post-training.