Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning
This work addresses a specific issue in knowledge distillation for machine learning practitioners, offering an incremental improvement over existing methods.
The paper tackles the problem of multi-teacher knowledge distillation where students with poor learning ability may not benefit from specialized integrated knowledge, proposing an adaptive method with meta-learning that leverages diverse teacher knowledge to enhance student performance, validated by experiments on multiple benchmark datasets.
Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources. Most existing methods explore different weighting strategies to obtain a powerful ensemble teacher, while ignoring the student with poor learning ability may not benefit from such specialized integrated knowledge. To address this problem, we propose Adaptive Multi-teacher Knowledge Distillation with Meta-Learning (MMKD) to supervise student with appropriate knowledge from a tailored ensemble teacher. With the help of a meta-weight network, the diverse yet compatible teacher knowledge in the output layer and intermediate layers is jointly leveraged to enhance the student performance. Extensive experiments on multiple benchmark datasets validate the effectiveness and flexibility of our methods. Code is available: https://github.com/Rorozhl/MMKD.