Embedding Compression for Teacher-to-Student Knowledge Transfer
This work addresses a specific bottleneck in knowledge distillation for cross-task scenarios, offering an incremental improvement for machine learning practitioners.
The paper tackles the problem of irrelevant knowledge in teacher embeddings for knowledge transfer across different tasks by proposing an embedding compression module with a trainable transformation, resulting in improved classification performance, particularly for unsupervised embeddings, and enhanced generalizability of student models.
Common knowledge distillation methods require the teacher model and the student model to be trained on the same task. However, the usage of embeddings as teachers has also been proposed for different source tasks and target tasks. Prior work that uses embeddings as teachers ignores the fact that the teacher embeddings are likely to contain irrelevant knowledge for the target task. To address this problem, we propose to use an embedding compression module with a trainable teacher transformation to obtain a compact teacher embedding. Results show that adding the embedding compression module improves the classification performance, especially for unsupervised teacher embeddings. Moreover, student models trained with the guidance of embeddings show stronger generalizability.