DropCluster: A structured dropout for convolutional networks
This addresses overfitting in convolutional networks for computer vision tasks, but it is incremental as it builds on existing structured Dropout methods.
The paper tackled the problem of Dropout being less effective in convolutional layers due to random feature dropping ignoring spatial correlations, by introducing DropCluster, a structured regularization method that clusters and drops features adaptively, resulting in better overfitting control on datasets like CIFAR-10/100, SVHN, and APPA-REAL.
Dropout as a common regularizer to prevent overfitting in deep neural networks has been less effective in convolutional layers than in fully connected layers. This is because Dropout drops features randomly, without considering local structure. When features are spatially correlated, as in the case of convolutional layers, information from the dropped features can still propagate to subsequent layers via neighboring features. To address this problem, structured forms of Dropout have been proposed. A drawback of these methods is that they do not adapt to the data. In this work, we leverage the structure in the outputs of convolutional layers and introduce a novel structured regularization method named DropCluster. Our approach clusters features in convolutional layers, and drops the resulting clusters randomly during training iterations. Experiments on CIFAR-10/100, SVHN, and APPA-REAL datasets demonstrate that our approach is effective and controls overfitting better than other approaches.