Towards Mitigating Architecture Overfitting on Distilled Datasets
This addresses a key bottleneck in dataset distillation for machine learning practitioners by improving generalization across architectures, though it is incremental as it builds on existing distillation techniques.
The paper tackles the problem of architecture overfitting in dataset distillation, where distilled datasets perform poorly when used with different network architectures, especially larger ones, and introduces methods like DropPath and knowledge distillation to mitigate this, achieving comparable or superior performance across various tasks and data sizes.
Dataset distillation methods have demonstrated remarkable performance for neural networks trained with very limited training data. However, a significant challenge arises in the form of \textit{architecture overfitting}: the distilled training dataset synthesized by a specific network architecture (i.e., training network) generates poor performance when trained by other network architectures (i.e., test networks), especially when the test networks have a larger capacity than the training network. This paper introduces a series of approaches to mitigate this issue. Among them, DropPath renders the large model to be an implicit ensemble of its sub-networks, and knowledge distillation ensures each sub-network acts similarly to the small but well-performing teacher network. These methods, characterized by their smoothing effects, significantly mitigate architecture overfitting. We conduct extensive experiments to demonstrate the effectiveness and generality of our methods. Particularly, across various scenarios involving different tasks and different sizes of distilled data, our approaches significantly mitigate architecture overfitting. Furthermore, our approaches achieve comparable or even superior performance when the test network is larger than the training network.