Dropout with Tabu Strategy for Regularizing Deep Neural Networks
This is an incremental improvement for researchers and practitioners using dropout to prevent overfitting in deep learning models.
The paper tackles the problem of improving dropout regularization in deep neural networks by introducing a diversification strategy called Tabu Dropout, which marks dropped units from the last forward propagation to encourage more varied neural network architectures, resulting in improved performance on MNIST and Fashion-MNIST datasets.
Dropout has proven to be an effective technique for regularization and preventing the co-adaptation of neurons in deep neural networks (DNN). It randomly drops units with a probability $p$ during the training stage of DNN. Dropout also provides a way of approximately combining exponentially many different neural network architectures efficiently. In this work, we add a diversification strategy into dropout, which aims at generating more different neural network architectures in a proper times of iterations. The dropped units in last forward propagation will be marked. Then the selected units for dropping in the current FP will be kept if they have been marked in the last forward propagation. We only mark the units from the last forward propagation. We call this new technique Tabu Dropout. Tabu Dropout has no extra parameters compared with the standard Dropout and also it is computationally cheap. The experiments conducted on MNIST, Fashion-MNIST datasets show that Tabu Dropout improves the performance of the standard dropout.