Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition
This work addresses a specific bottleneck in multi-teacher knowledge distillation for visual recognition, offering an incremental improvement over existing methods.
The paper tackles the problem of balancing distillation strengths among multiple teachers in knowledge distillation by proposing MTKD-RL, which uses reinforcement learning to optimize teacher weights based on performance and teacher-student gaps, achieving state-of-the-art results on visual recognition tasks.
Multi-teacher Knowledge Distillation (KD) transfers diverse knowledge from a teacher pool to a student network. The core problem of multi-teacher KD is how to balance distillation strengths among various teachers. Most existing methods often develop weighting strategies from an individual perspective of teacher performance or teacher-student gaps, lacking comprehensive information for guidance. This paper proposes Multi-Teacher Knowledge Distillation with Reinforcement Learning (MTKD-RL) to optimize multi-teacher weights. In this framework, we construct both teacher performance and teacher-student gaps as state information to an agent. The agent outputs the teacher weight and can be updated by the return reward from the student. MTKD-RL reinforces the interaction between the student and teacher using an agent in an RL-based decision mechanism, achieving better matching capability with more meaningful weights. Experimental results on visual recognition tasks, including image classification, object detection, and semantic segmentation tasks, demonstrate that MTKD-RL achieves state-of-the-art performance compared to the existing multi-teacher KD works.