Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
For practitioners building compact instruction-following models, this work provides practical guidance on when and how to use KD effectively in data-scarce environments.
This paper systematically studies knowledge distillation (KD) in post-training for large language models, finding that KD outperforms supervised fine-tuning in low-data regimes but its advantage diminishes with more data; distilling from a stronger teacher restores gains even with abundant data. A two-stage KD strategy using synthetic teacher-labeled data followed by human annotation refinement consistently improves student performance.
Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.