Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation
This work addresses model compression for neural machine translation, offering insights into SLKD mechanisms, but it is incremental as it builds on existing distillation techniques.
The paper investigated why sequence-level knowledge distillation (SLKD) helps train smaller neural machine translation models, finding that it likely acts as data augmentation rather than simplifying noisy data, and achieved BLEU gains of 0.7-1.2 on TED Talks.
Sequence-level knowledge distillation (SLKD) is a model compression technique that leverages large, accurate teacher models to train smaller, under-parameterized student models. Why does pre-processing MT data with SLKD help us train smaller models? We test the common hypothesis that SLKD addresses a capacity deficiency in students by "simplifying" noisy data points and find it unlikely in our case. Models trained on concatenations of original and "simplified" datasets generalize just as well as baseline SLKD. We then propose an alternative hypothesis under the lens of data augmentation and regularization. We try various augmentation strategies and observe that dropout regularization can become unnecessary. Our methods achieve BLEU gains of 0.7-1.2 on TED Talks.