MetaMixer: A Regularization Strategy for Online Knowledge Distillation
This addresses a gap in online knowledge distillation for improving model localization and overall image understanding, though it appears incremental as it builds on existing methods.
The paper tackled the problem of overlooking multi-level knowledge, especially low-level knowledge, in online knowledge distillation by proposing MetaMixer, a regularization strategy that combines low-level and high-level knowledge, resulting in significant performance gains over state-of-the-art methods.
Online knowledge distillation (KD) has received increasing attention in recent years. However, while most existing online KD methods focus on developing complicated model structures and training strategies to improve the distillation of high-level knowledge like probability distribution, the effects of the multi-level knowledge in the online KD are greatly overlooked, especially the low-level knowledge. Thus, to provide a novel viewpoint to online KD, we propose MetaMixer, a regularization strategy that can strengthen the distillation by combining the low-level knowledge that impacts the localization capability of the networks, and high-level knowledge that focuses on the whole image. Experiments under different conditions show that MetaMixer can achieve significant performance gains over state-of-the-art methods.