One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning
This work addresses the problem of real-time training efficiency for large-scale deep learning systems in production, offering an incremental improvement over existing ad-hoc sampling methods.
The paper tackles the challenge of efficiently selecting training data from large-scale streaming environments by leveraging information from forward passes during inference, resulting in improved data selection for training.
Deep learning models in large-scale machine learning systems are often continuously trained with enormous data from production environments. The sheer volume of streaming training data poses a significant challenge to real-time training subsystems and ad-hoc sampling is the standard practice. Our key insight is that these deployed ML systems continuously perform forward passes on data instances during inference, but ad-hoc sampling does not take advantage of this substantial computational effort. Therefore, we propose to record a constant amount of information per instance from these forward passes. The extra information measurably improves the selection of which data instances should participate in forward and backward passes. A novel optimization framework is proposed to analyze this problem and we provide an efficient approximation algorithm under the framework of Mini-batch gradient descent as a practical solution. We also demonstrate the effectiveness of our framework and algorithm on several large-scale classification and regression tasks, when compared with competitive baselines widely used in industry.