Dropout is a special case of the stochastic delta rule: faster and more accurate deep learning
This work addresses overfitting and training efficiency in deep learning, offering a faster and more accurate alternative to Dropout, though it is incremental as it builds on an existing method.
The paper tackles the problem of overfitting and slow convergence in deep learning by showing that Dropout is a special case of the Stochastic Delta Rule (SDR), a more general method that models weights as random variables updated based on prediction error. The result is that SDR outperforms Dropout by approximately 17% in test error on CIFAR-100 with DenseNet-BC 250 and reduces training time from 100 to 35 epochs for the same accuracy.
Multi-layer neural networks have lead to remarkable performance on many kinds of benchmark tasks in text, speech and image processing. Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification. One approach to these estimation and related problems (local minima, colinearity, feature discovery etc.) is called Dropout (Hinton, et al 2012, Baldi et al 2016). The Dropout algorithm removes hidden units according to a Bernoulli random variable with probability $p$ prior to each update, creating random "shocks" to the network that are averaged over updates. In this paper we will show that Dropout is a special case of a more general model published originally in 1990 called the Stochastic Delta Rule, or SDR (Hanson, 1990). SDR redefines each weight in the network as a random variable with mean $μ_{w_{ij}}$ and standard deviation $σ_{w_{ij}}$. Each weight random variable is sampled on each forward activation, consequently creating an exponential number of potential networks with shared weights. Both parameters are updated according to prediction error, thus resulting in weight noise injections that reflect a local history of prediction error and local model averaging. SDR therefore implements a more sensitive local gradient-dependent simulated annealing per weight converging in the limit to a Bayes optimal network. Tests on standard benchmarks (CIFAR) using a modified version of DenseNet shows the SDR outperforms standard Dropout in test error by approx. $17\%$ with DenseNet-BC 250 on CIFAR-100 and approx. $12-14\%$ in smaller networks. We also show that SDR reaches the same accuracy that Dropout attains in 100 epochs in as few as 35 epochs.