Learning Interpretation with Explainable Knowledge Distillation
This addresses the need for more interpretable and consistent model compression in AI, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem that existing knowledge distillation methods fail to transfer explainable knowledge from teacher to student models, leading to inconsistent explanations. It proposes XDistillation, which uses convolutional autoencoders to approximate teacher explanations, resulting in improved predictive accuracy and faithfulness compared to conventional methods.
Knowledge Distillation (KD) has been considered as a key solution in model compression and acceleration in recent years. In KD, a small student model is generally trained from a large teacher model by minimizing the divergence between the probabilistic outputs of the two. However, as demonstrated in our experiments, existing KD methods might not transfer critical explainable knowledge of the teacher to the student, i.e. the explanations of predictions made by the two models are not consistent. In this paper, we propose a novel explainable knowledge distillation model, called XDistillation, through which both the performance the explanations' information are transferred from the teacher model to the student model. The XDistillation model leverages the idea of convolutional autoencoders to approximate the teacher explanations. Our experiments shows that models trained by XDistillation outperform those trained by conventional KD methods not only in term of predictive accuracy but also faithfulness to the teacher models.