Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
This addresses the problem of expensive dataset pruning for deep learning practitioners by offering a more efficient approach, though it is incremental as it builds on existing pruning methods.
The paper tackles the high cost of dataset pruning by introducing a lightweight method that uses example difficulty and prediction uncertainty to identify important samples early, reducing pruning time to 66% on ImageNet-1k and 15% on CIFAR datasets while achieving state-of-the-art accuracy, such as 60% test accuracy at a 90% pruning ratio.
Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning, we further propose a ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66% compared to previous methods while achieving a SOTA, specifically 60% test accuracy at a 90% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15% while maintaining SOTA performance.