Variance Suppression: Balanced Training Process in Deep Learning
This addresses training inefficiencies in deep learning for practitioners, but it appears incremental as it modifies gradient descent rather than introducing a new paradigm.
The paper tackles the problem of unbalanced training in deep learning caused by stochastic gradient descent on unbalanced data, resulting in VSSGD, which accelerates training, prevents overfitting, and improves learning from small samples.
Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obtained is unbalanced. To address this issue, this paper takes the error variance and error mean both into consideration. The adaptively adjusting approach of two terms trading off is also given in our algorithm. Due to this algorithm can suppress error variance, we named it Variance Suppression Gradient Descent (VSSGD). Experimental results have demonstrated that VSSGD can accelerate the training process, effectively prevent overfitting, improve the networks learning capacity from small samples.