Cross-Image Relational Knowledge Distillation for Semantic Segmentation
This work addresses a bottleneck in semantic segmentation for computer vision applications, offering an incremental improvement over existing distillation methods.
The paper tackles the problem of knowledge distillation for semantic segmentation by proposing a method that transfers global pixel relations across images, improving segmentation performance on Cityscapes, CamVid, and Pascal VOC datasets.
Current Knowledge Distillation (KD) methods for semantic segmentation often guide the student to mimic the teacher's structured information generated from individual data samples. However, they ignore the global semantic relations among pixels across various images that are valuable for KD. This paper proposes a novel Cross-Image Relational KD (CIRKD), which focuses on transferring structured pixel-to-pixel and pixel-to-region relations among the whole images. The motivation is that a good teacher network could construct a well-structured feature space in terms of global pixel dependencies. CIRKD makes the student mimic better structured semantic relations from the teacher, thus improving the segmentation performance. Experimental results over Cityscapes, CamVid and Pascal VOC datasets demonstrate the effectiveness of our proposed approach against state-of-the-art distillation methods. The code is available at https://github.com/winycg/CIRKD.