Sparse Teachers Can Be Dense with Knowledge
This work addresses a bottleneck in knowledge distillation for language models, offering an incremental improvement for practitioners in NLP.
The paper tackles the problem of distilling knowledge from over-parameterized teachers to students by identifying that such teachers can produce student-unfriendly knowledge, limiting overall effectiveness. It proposes a sparse teacher trick to remove unfriendly parameters, resulting in students with compelling performance on the GLUE benchmark compared to competitive baselines.
Recent advances in distilling pretrained language models have discovered that, besides the expressiveness of knowledge, the student-friendliness should be taken into consideration to realize a truly knowledgable teacher. Based on a pilot study, we find that over-parameterized teachers can produce expressive yet student-unfriendly knowledge and are thus limited in overall knowledgableness. To remove the parameters that result in student-unfriendliness, we propose a sparse teacher trick under the guidance of an overall knowledgable score for each teacher parameter. The knowledgable score is essentially an interpolation of the expressiveness and student-friendliness scores. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Extensive experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance in comparison with a series of competitive baselines.