Efficient batchwise dropout training using submatrices
This is an incremental improvement for practitioners training neural networks with dropout.
The paper tackles the computational inefficiency of standard dropout training by proposing batchwise dropout, which applies one dropout pattern per minibatch instead of per sample, reducing training times substantially.
Dropout is a popular technique for regularizing artificial neural networks. Dropout networks are generally trained by minibatch gradient descent with a dropout mask turning off some of the units---a different pattern of dropout is applied to every sample in the minibatch. We explore a very simple alternative to the dropout mask. Instead of masking dropped out units by setting them to zero, we perform matrix multiplication using a submatrix of the weight matrix---unneeded hidden units are never calculated. Performing dropout batchwise, so that one pattern of dropout is used for each sample in a minibatch, we can substantially reduce training times. Batchwise dropout can be used with fully-connected and convolutional neural networks.