Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights
This work addresses the problem of efficient deployment of small language models for real-life applications, representing an incremental advancement in knowledge distillation techniques.
The paper tackles the challenge of deploying small language models by introducing a knowledge distillation method that uses a teacher model to identify influential tokens as rationales for the student model, showing improvements over standard fine-tuning and state-of-the-art distillation models on four datasets.
Enhancing small language models for real-life application deployment is a significant challenge facing the research community. Due to the difficulties and costs of using large language models, researchers are seeking ways to effectively deploy task-specific small models. In this work, we introduce a simple yet effective knowledge distillation method to improve the performance of small language models. Our approach utilizes a teacher model with approximately 3 billion parameters to identify the most influential tokens in its decision-making process. These tokens are extracted from the input based on their attribution scores relative to the output, using methods like saliency maps. These important tokens are then provided as rationales to a student model, aiming to distill the knowledge of the teacher model. This method has proven to be effective, as demonstrated by testing it on four diverse datasets, where it shows improvement over both standard fine-tuning methods and state-of-the-art knowledge distillation models. Furthermore, we explore explanations of the success of the model by analyzing the important tokens extracted from the teacher model. Our findings reveal that in 68\% of cases, specifically in datasets where labels are part of the answer, such as multiple-choice questions, the extracted tokens are part of the ground truth.