Soft Random Sampling: A Theoretical and Empirical Analysis
This addresses the computational bottleneck in training deep networks on massive data for applications like image and speech recognition, though it is incremental as it builds on existing sampling techniques.
The paper tackles the problem of efficient training of large-scale deep neural networks with massive data by analyzing Soft Random Sampling (SRS), showing it offers a better accuracy-efficiency trade-off than coreset-based methods, with significant speedup and competitive performance on industrial datasets.
Soft random sampling (SRS) is a simple yet effective approach for efficient training of large-scale deep neural networks when dealing with massive data. SRS selects a subset uniformly at random with replacement from the full data set in each epoch. In this paper, we conduct a theoretical and empirical analysis of SRS. First, we analyze its sampling dynamics including data coverage and occupancy. Next, we investigate its convergence with non-convex objective functions and give the convergence rate. Finally, we provide its generalization performance. We empirically evaluate SRS for image recognition on CIFAR10 and automatic speech recognition on Librispeech and an in-house payload dataset to demonstrate its effectiveness. Compared to existing coreset-based data selection methods, SRS offers a better accuracy-efficiency trade-off. Especially on real-world industrial scale data sets, it is shown to be a powerful training strategy with significant speedup and competitive performance with almost no additional computing cost.