FocusedDropout for Convolutional Neural Network
This addresses the issue of dropout inefficiency in CNNs for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackles the problem of dropout being ineffective in convolutional neural networks due to spatially correlated features by proposing FocusedDropout, a non-random method that retains target-related features and discards others, resulting in improved performance across multiple datasets like CIFAR10 and CIFAR100 with a 10% batch usage cost.
In convolutional neural network (CNN), dropout cannot work well because dropped information is not entirely obscured in convolutional layers where features are correlated spatially. Except randomly discarding regions or channels, many approaches try to overcome this defect by dropping influential units. In this paper, we propose a non-random dropout method named FocusedDropout, aiming to make the network focus more on the target. In FocusedDropout, we use a simple but effective way to search for the target-related features, retain these features and discard others, which is contrary to the existing methods. We found that this novel method can improve network performance by making the network more target-focused. Besides, increasing the weight decay while using FocusedDropout can avoid the overfitting and increase accuracy. Experimental results show that even a slight cost, 10\% of batches employing FocusedDropout, can produce a nice performance boost over the baselines on multiple datasets of classification, including CIFAR10, CIFAR100, Tiny Imagenet, and has a good versatility for different CNN models.