Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images
This work addresses model compression for image restoration, an underexplored area, but it is incremental as it adapts knowledge distillation techniques to a specific domain.
The authors tackled model compression for image restoration by proposing a Simultaneous Learning Knowledge Distillation (SLKD) framework, which achieved over 80% reductions in FLOPs and parameters while maintaining strong performance across five datasets and three tasks.
Model compression through knowledge distillation has seen extensive application in classification and segmentation tasks. However, its potential in image-to-image translation, particularly in image restoration, remains underexplored. To address this gap, we propose a Simultaneous Learning Knowledge Distillation (SLKD) framework tailored for model compression in image restoration tasks. SLKD employs a dual-teacher, single-student architecture with two distinct learning strategies: Degradation Removal Learning (DRL) and Image Reconstruction Learning (IRL), simultaneously. In DRL, the student encoder learns from Teacher A to focus on removing degradation factors, guided by a novel BRISQUE extractor. In IRL, the student decoder learns from Teacher B to reconstruct clean images, with the assistance of a proposed PIQE extractor. These strategies enable the student to learn from degraded and clean images simultaneously, ensuring high-quality compression of image restoration models. Experimental results across five datasets and three tasks demonstrate that SLKD achieves substantial reductions in FLOPs and parameters, exceeding 80\%, while maintaining strong image restoration performance.