Improve Cross-Architecture Generalization on Dataset Distillation
This work addresses a bottleneck in dataset distillation for machine learning practitioners by improving cross-architecture generalization, though it appears incremental as it builds on existing distillation methods.
The paper tackles the problem of model-specific biases limiting generalization in dataset distillation by proposing a 'model pool' approach that selects models from a diverse pool during distillation and integrates knowledge distillation in testing, resulting in validated superior performance across various models.
Dataset distillation, a pragmatic approach in machine learning, aims to create a smaller synthetic dataset from a larger existing dataset. However, existing distillation methods primarily adopt a model-based paradigm, where the synthetic dataset inherits model-specific biases, limiting its generalizability to alternative models. In response to this constraint, we propose a novel methodology termed "model pool". This approach involves selecting models from a diverse model pool based on a specific probability distribution during the data distillation process. Additionally, we integrate our model pool with the established knowledge distillation approach and apply knowledge distillation to the test process of the distilled dataset. Our experimental results validate the effectiveness of the model pool approach across a range of existing models while testing, demonstrating superior performance compared to existing methodologies.