Adaptive Tabu Dropout for Regularization of Deep Neural Network
This work addresses regularization for deep learning practitioners, but it is incremental as it builds upon existing Tabu Dropout mechanisms.
The paper tackled improving regularization in deep neural networks by enhancing Tabu Dropout with adaptive tabu tenure, resulting in significantly better performance on standard benchmark datasets compared to standard and basic tabu dropout methods.
Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. In this paper, we improve the Tabu Dropout mechanism for training deep neural networks in two ways. Firstly, we propose to use tabu tenure, or the number of epochs a particular unit will not be dropped. Different tabu tenures provide diversification to boost the training of deep neural networks based on the search landscape. Secondly, we propose an adaptive tabu algorithm that automatically selects the tabu tenure based on the training performances through epochs. On several standard benchmark datasets, the experimental results show that the adaptive tabu dropout and tabu tenure dropout diversify and perform significantly better compared to the standard dropout and basic tabu dropout mechanisms.