Transferring Inter-Class Correlation
This work addresses a specific bottleneck in knowledge distillation for classification tasks, representing an incremental improvement.
The paper tackles the problem of defining efficient knowledge transfer in the Teacher-Student framework for classification by proposing a novel Self-Attention based Inter-Class Correlation map, resulting in the ICCT method.
The Teacher-Student (T-S) framework is widely utilized in the classification tasks, through which the performance of one neural network (the student) can be improved by transferring knowledge from another trained neural network (the teacher). Since the transferring knowledge is related to the network capacities and structures between the teacher and the student, how to define efficient knowledge remains an open question. To address this issue, we design a novel transferring knowledge, the Self-Attention based Inter-Class Correlation (ICC) map in the output layer, and propose our T-S framework, Inter-Class Correlation Transfer (ICCT).