Don't Throw Away Data: Better Sequence Knowledge Distillation
This work addresses a specific bottleneck in knowledge distillation for machine translation, offering incremental improvements over existing methods.
The paper tackled the problem of improving sequence knowledge distillation by integrating minimum Bayes risk (MBR) decoding more tightly, using multiple high-scoring translations instead of a single sequence, and achieved consistent improvements in English to German and English to Japanese translation tasks across varying model sizes.
A critical component in knowledge distillation is the means of coupling the teacher and student. The predominant sequence knowledge distillation method involves supervised learning of the student against teacher-decoded outputs, and is exemplified by the current state of the art, which incorporates minimum Bayes risk (MBR) decoding. In this paper we seek to integrate MBR more tightly in distillation training, specifically by using several high scoring MBR translations, rather than a single selected sequence, thus capturing a rich diversity of teacher outputs. Our experiments on English to German and English to Japanese translation show consistent improvements over strong baseline methods for both tasks and with varying model sizes. Additionally, we conduct a detailed analysis focusing on data efficiency and capacity curse aspects to elucidate MBR-n and explore its further potential.