AdaSelection: Accelerating Deep Learning Training through Data Subsampling
This addresses the problem of inefficient data usage in large-scale deep learning training for practitioners, though it appears incremental as it builds on existing subsampling methods.
The paper tackles the problem of accelerating deep learning training by introducing AdaSelection, an adaptive subsampling method that identifies informative sub-samples within minibatches, and it demonstrates superior performance compared to industry-standard baselines across various tasks without sacrificing model performance.
In this paper, we introduce AdaSelection, an adaptive sub-sampling method to identify the most informative sub-samples within each minibatch to speed up the training of large-scale deep learning models without sacrificing model performance. Our method is able to flexibly combines an arbitrary number of baseline sub-sampling methods incorporating the method-level importance and intra-method sample-level importance at each iteration. The standard practice of ad-hoc sampling often leads to continuous training with vast amounts of data from production environments. To improve the selection of data instances during forward and backward passes, we propose recording a constant amount of information per instance from these passes. We demonstrate the effectiveness of our method by testing it across various types of inputs and tasks, including the classification tasks on both image and language datasets, as well as regression tasks. Compared with industry-standard baselines, AdaSelection consistently displays superior performance.