Evolving Knowledge Distillation for Lightweight Neural Machine Translation
For practitioners deploying NMT on resource-limited devices, EKD provides a method to compress large models while maintaining high translation quality, addressing a known bottleneck in knowledge distillation.
The paper tackles the capacity gap problem in knowledge distillation for neural machine translation, where large teacher models cannot effectively teach small student models. The proposed Evolving Knowledge Distillation (EKD) uses a sequence of teachers with increasing capacities, enabling a compact student to achieve a BLEU score of 34.24 on IWSLT-14, within 0.08 BLEU of the strongest teacher.
Recent advancements in Neural Machine Translation (NMT) have significantly improved translation quality. However, the increasing size and complexity of state-of-the-art models present significant challenges for deployment on resource-limited devices. Knowledge distillation (KD) is a promising approach for compressing models, but its effectiveness diminishes when there is a large capacity gap between teacher and student models. To address this issue, we propose Evolving Knowledge Distillation (EKD), a progressive training framework in which the student model learns from a sequence of teachers with gradually increasing capacities. Experiments on IWSLT-14, WMT-17, and WMT-23 benchmarks show that EKD leads to consistent improvements at each stage. On IWSLT-14, the final student achieves a BLEU score of 34.24, narrowing the gap to the strongest teacher (34.32 BLEU) to just 0.08 BLEU. Similar trends are observed on other datasets. These results demonstrate that EKD effectively bridges the capacity gap, enabling compact models to achieve performance close to that of much larger teacher models.Code and models are available at https://github.com/agi-content-generation/EKD.