Exploring Variance Reduction in Importance Sampling for Efficient DNN Training
This addresses computational bottlenecks in DNN training for researchers and practitioners, though it appears incremental as it builds on existing importance sampling methods.
The paper tackles the challenge of efficiently assessing variance reduction in importance sampling for DNN training, proposing a method that uses minibatches to estimate variance reduction and enable automatic learning rate adjustment, with experiments showing consistent variance reduction, improved training efficiency, and enhanced model accuracy compared to existing approaches.
Importance sampling is widely used to improve the efficiency of deep neural network (DNN) training by reducing the variance of gradient estimators. However, efficiently assessing the variance reduction relative to uniform sampling remains challenging due to computational overhead. This paper proposes a method for estimating variance reduction during DNN training using only minibatches sampled under importance sampling. By leveraging the proposed method, the paper also proposes an effective minibatch size to enable automatic learning rate adjustment. An absolute metric to quantify the efficiency of importance sampling is also introduced as well as an algorithm for real-time estimation of importance scores based on moving gradient statistics. Theoretical analysis and experiments on benchmark datasets demonstrated that the proposed algorithm consistently reduces variance, improves training efficiency, and enhances model accuracy compared with current importance-sampling approaches while maintaining minimal computational overhead.