Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers
This addresses memory constraints for edge deployment in machine translation, but it is incremental as it builds on existing knowledge distillation methods.
The paper tackles the problem of deploying large neural machine translation models on edge devices by proposing a knowledge distillation technique that uses combinatorial layer-level supervision, achieving comparable results with 50% fewer parameters in low-resource settings.
With the growth of computing power neural machine translation (NMT) models also grow accordingly and become better. However, they also become harder to deploy on edge devices due to memory constraints. To cope with this problem, a common practice is to distill knowledge from a large and accurately-trained teacher network (T) into a compact student network (S). Although knowledge distillation (KD) is useful in most cases, our study shows that existing KD techniques might not be suitable enough for deep NMT engines, so we propose a novel alternative. In our model, besides matching T and S predictions we have a combinatorial mechanism to inject layer-level supervision from T to S. In this paper, we target low-resource settings and evaluate our translation engines for Portuguese--English, Turkish--English, and English--German directions. Students trained using our technique have 50% fewer parameters and can still deliver comparable results to those of 12-layer teachers.