What is Lost in Knowledge Distillation?
It addresses the problem of costly training and maintenance of deep neural networks for NLP practitioners by analyzing knowledge distillation losses, though it is incremental as it builds on existing compression techniques.
This paper investigates information losses in knowledge distillation for NLP models, finding that the distillation process is lossy and identifying patterns in how factors like layers and attention heads affect task sensitivity, with results providing data points to guide efficient configurations.
Deep neural networks (DNNs) have improved NLP tasks significantly, but training and maintaining such networks could be costly. Model compression techniques, such as, knowledge distillation (KD), have been proposed to address the issue; however, the compression process could be lossy. Motivated by this, our work investigates how a distilled student model differs from its teacher, if the distillation process causes any information losses, and if the loss follows a specific pattern. Our experiments aim to shed light on the type of tasks might be less or more sensitive to KD by reporting data points on the contribution of different factors, such as the number of layers or attention heads. Results such as ours could be utilized when determining effective and efficient configurations to achieve optimal information transfers between larger (teacher) and smaller (student) models.