Generative Dataset Distillation Based on Self-knowledge Distillation
This addresses the problem of reducing training costs and complexity in machine learning through more efficient dataset compression, though it appears incremental as it builds on existing distillation techniques.
The paper tackles dataset distillation by proposing a generative method that integrates self-knowledge distillation and logit standardization to improve alignment accuracy between synthetic and original data, achieving superior performance over state-of-the-art methods in experiments.
Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.