AutoSampling: Search for Effective Data Sampling Schedules
This work addresses the challenge of optimizing data sampling schedules for deep learning practitioners, offering a novel automated approach that can enhance training efficiency and model accuracy in image classification.
The paper tackles the problem of learning effective data sampling schedules for deep learning model training, which is difficult due to high-dimensional parameters, by proposing AutoSampling, a method that combines multi-exploitation and exploration steps to automatically learn robust schedules, achieving improved performance on various image classification tasks.
Data sampling acts as a pivotal role in training deep learning models. However, an effective sampling schedule is difficult to learn due to the inherently high dimension of parameters in learning the sampling schedule. In this paper, we propose an AutoSampling method to automatically learn sampling schedules for model training, which consists of the multi-exploitation step aiming for optimal local sampling schedules and the exploration step for the ideal sampling distribution. More specifically, we achieve sampling schedule search with shortened exploitation cycle to provide enough supervision. In addition, we periodically estimate the sampling distribution from the learned sampling schedules and perturb it to search in the distribution space. The combination of two searches allows us to learn a robust sampling schedule. We apply our AutoSampling method to a variety of image classification tasks illustrating the effectiveness of the proposed method.