Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration
This work addresses the challenge of enhancing compact student networks for image restoration, which is important for applications requiring efficient deployment, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of inefficient knowledge distillation in image restoration by proposing a dynamic contrastive framework that adjusts the solution space based on the student's learning state and aligns pixel-level distributions, resulting in significant performance improvements over state-of-the-art methods across various tasks and backbones.
Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student's learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones.