On Implicit Filter Level Sparsity in Convolutional Neural Networks
This work addresses the problem of optimizing CNN training and efficiency for machine learning practitioners, offering insights into sparsity mechanisms and practical speedup strategies, though it is incremental in building on existing training techniques.
The paper investigates the implicit filter-level sparsity that arises in CNNs trained with Batch Normalization, ReLU, adaptive gradient descent, and L2 regularization, analyzing its mechanisms and impact on performance gaps between adaptive and non-adaptive methods. It shows that this sparsity can be leveraged for neural network speedup, achieving performance comparable to or better than explicit pruning approaches without modifying the training pipeline.
We investigate filter level sparsity that emerges in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay. We conduct an extensive experimental study casting our initial findings into hypotheses and conclusions about the mechanisms underlying the emergent filter level sparsity. This study allows new insight into the performance gap obeserved between adapative and non-adaptive gradient descent methods in practice. Further, analysis of the effect of training strategies and hyperparameters on the sparsity leads to practical suggestions in designing CNN training strategies enabling us to explore the tradeoffs between feature selectivity, network capacity, and generalization performance. Lastly, we show that the implicit sparsity can be harnessed for neural network speedup at par or better than explicit sparsification / pruning approaches, with no modifications to the typical training pipeline required.