Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement
This work addresses the challenge of dataset quality for machine learning practitioners by providing a one-time dataset improvement method that benefits multiple model architectures, though it is incremental as it builds on existing techniques like data augmentation and distillation.
The paper tackles the problem of improving model accuracy and robustness by enhancing datasets, proposing Dataset Reinforcement, which uses data augmentation and knowledge distillation to create reinforced datasets like ImageNet+ that boost performance across various models and tasks without extra training cost. Results include a 1.7% accuracy gain for ResNet-50 on ImageNet, up to 20% robustness improvements on ImageNet-R/A/C, and enhanced transfer learning with up to 3.4% accuracy gains on downstream datasets.
We propose Dataset Reinforcement, a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no additional training cost for users. We propose a Dataset Reinforcement strategy based on data augmentation and knowledge distillation. Our generic strategy is designed based on extensive analysis across CNN- and transformer-based models and performing large-scale study of distillation with state-of-the-art models with various data augmentations. We create a reinforced version of the ImageNet training dataset, called ImageNet+, as well as reinforced datasets CIFAR-100+, Flowers-102+, and Food-101+. Models trained with ImageNet+ are more accurate, robust, and calibrated, and transfer well to downstream tasks (e.g., segmentation and detection). As an example, the accuracy of ResNet-50 improves by 1.7% on the ImageNet validation set, 3.5% on ImageNetV2, and 10.0% on ImageNet-R. Expected Calibration Error (ECE) on the ImageNet validation set is also reduced by 9.9%. Using this backbone with Mask-RCNN for object detection on MS-COCO, the mean average precision improves by 0.8%. We reach similar gains for MobileNets, ViTs, and Swin-Transformers. For MobileNetV3 and Swin-Tiny, we observe significant improvements on ImageNet-R/A/C of up to 20% improved robustness. Models pretrained on ImageNet+ and fine-tuned on CIFAR-100+, Flowers-102+, and Food-101+, reach up to 3.4% improved accuracy. The code, datasets, and pretrained models are available at https://github.com/apple/ml-dr.