EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer
For practitioners deploying smaller LLMs, this method improves distillation efficiency by focusing on informative tokens, but the gains are incremental over existing token-level approaches.
The paper tackles inefficient token-level knowledge distillation in LLMs by proposing an entropy-guided adaptive strategy that dynamically adjusts training focus, temperature, and architecture per token. Experiments show improved student performance over uniform distillation baselines.
Large language models (LLMs) have achieved remarkable performance across diverse domains, yet their enormous computational and memory requirements hinder deployment in resource-constrained environments. Knowledge distillation offers a promising solution by transferring knowledge from a large teacher model to a smaller student model. However, existing distillation methods typically treat all tokens equally, ignoring the fact that different tokens contribute unequally to model decisions. This can lead to inefficient knowledge transfer and reduced learning effectiveness. To address this limitation, we propose an entropy-based adaptive distillation strategy that dynamically adjusts the training process at the token level. Our method leverages the teacher's output entropy to guide three aspects of distillation. Specifically, we introduce a token-level curriculum by dynamically shifting focus from low- to high-entropy tokens during training. We further adjust the distillation temperature based on token entropy to better capture teacher confidence patterns. Moreover, we employ a dual-branch architecture for efficient logits-only distillation on easy tokens and deeper feature-based distillation on difficult tokens. Extensive experiments validate the soundness and effectiveness of our method.