CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation
This addresses a bottleneck in DFKD for machine learning practitioners by enhancing model generalization without real data, though it is incremental as it builds on existing DFKD methods.
The paper tackles the problem of limited transferability in Data-Free Knowledge Distillation (DFKD) by proposing CAE-DFKD, which improves generalization at the embedding level, achieving competitive performance on image recognition and enhanced transferability in downstream tasks.
Data-Free Knowledge Distillation (DFKD) enables the knowledge transfer from the given pre-trained teacher network to the target student model without access to the real training data. Existing DFKD methods focus primarily on improving image recognition performance on associated datasets, often neglecting the crucial aspect of the transferability of learned representations. In this paper, we propose Category-Aware Embedding Data-Free Knowledge Distillation (CAE-DFKD), which addresses at the embedding level the limitations of previous rely on image-level methods to improve model generalization but fail when directly applied to DFKD. The superiority and flexibility of CAE-DFKD are extensively evaluated, including: \textit{\textbf{i.)}} Significant efficiency advantages resulting from altering the generator training paradigm; \textit{\textbf{ii.)}} Competitive performance with existing DFKD state-of-the-art methods on image recognition tasks; \textit{\textbf{iii.)}} Remarkable transferability of data-free learned representations demonstrated in downstream tasks.