Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
This work provides a comprehensive overview for researchers and practitioners, but it is incremental as it surveys existing methods without introducing new techniques.
This survey tackles the problem of transferring knowledge from complex teacher models to simpler student models in knowledge distillation, synthesizing recent literature to highlight advancements in methods like attention-based approaches and their effectiveness in compressing large language models while preserving accuracy.
Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy. It has demonstrated substantial advancements in various applications including image classification, object detection, language modeling, text classification, and sentiment analysis. Recent innovations in KD methods, such as attention-based approaches, block-wise logit distillation, and decoupling distillation, have notably improved student model performance. These techniques focus on stimulus complexity, attention mechanisms, and global information capture to optimize knowledge transfer. In addition, KD has proven effective in compressing large language models while preserving accuracy, reducing computational overhead, and improving inference speed. This survey synthesizes the latest literature, highlighting key findings, contributions, and future directions in knowledge distillation to provide insights for researchers and practitioners on its evolving role in artificial intelligence and machine learning.