Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation
This provides a practical solution for deploying high-performance language models in resource-constrained edge environments, though it is incremental as it builds on existing knowledge distillation and fine-tuning methods.
The paper tackles the problem of small language models underperforming after pre-training by introducing a systematic post-training pipeline, which achieves state-of-the-art performance among billion-parameter models with strong generalization on edge devices.
The rapid advancement of large language models (LLMs) has significantly advanced the capabilities of artificial intelligence across various domains. However, their massive scale and high computational costs render them unsuitable for direct deployment in resource-constrained edge environments. This creates a critical need for high-performance small models that can operate efficiently at the edge. Yet, after pre-training alone, these smaller models often fail to meet the performance requirements of complex tasks. To bridge this gap, we introduce a systematic post-training pipeline that efficiently enhances small model accuracy. Our post training pipeline consists of curriculum-based supervised fine-tuning (SFT) and offline on-policy knowledge distillation. The resulting instruction-tuned model achieves state-of-the-art performance among billion-parameter models, demonstrating strong generalization under strict hardware constraints while maintaining competitive accuracy across a variety of tasks. This work provides a practical and efficient solution for developing high-performance language models on Ascend edge devices.