Knowledge Distillation from Single to Multi Labels: an Empirical Study
This work addresses the problem of applying knowledge distillation to multi-label classification, which is incremental as it adapts existing techniques to a less-explored domain.
The study investigated knowledge distillation for multi-label classification, finding that existing logit-based and feature-based methods are ineffective, and proposed a new method based on Class Activation Maps (CAMs) that consistently outperforms other approaches across various settings.
Knowledge distillation (KD) has been extensively studied in single-label image classification. However, its efficacy for multi-label classification remains relatively unexplored. In this study, we firstly investigate the effectiveness of classical KD techniques, including logit-based and feature-based methods, for multi-label classification. Our findings indicate that the logit-based method is not well-suited for multi-label classification, as the teacher fails to provide inter-category similarity information or regularization effect on student model's training. Moreover, we observe that feature-based methods struggle to convey compact information of multiple labels simultaneously. Given these limitations, we propose that a suitable dark knowledge should incorporate class-wise information and be highly correlated with the final classification results. To address these issues, we introduce a novel distillation method based on Class Activation Maps (CAMs), which is both effective and straightforward to implement. Across a wide range of settings, CAMs-based distillation consistently outperforms other methods.