Feather: An Elegant Solution to Effective DNN Sparsification
This addresses the need for compact and efficient deep neural networks in resource-limited environments, representing a strong specific gain rather than a foundational advancement.
The paper tackles the problem of neural network pruning for efficient models by introducing Feather, a sparse training module that achieves state-of-the-art Top-1 validation accuracy on ImageNet with ResNet-50, surpassing existing methods by a considerable margin.
Neural Network pruning is an increasingly popular way for producing compact and efficient models, suitable for resource-limited environments, while preserving high performance. While the pruning can be performed using a multi-cycle training and fine-tuning process, the recent trend is to encompass the sparsification process during the standard course of training. To this end, we introduce Feather, an efficient sparse training module utilizing the powerful Straight-Through Estimator as its core, coupled with a new thresholding operator and a gradient scaling technique, enabling robust, out-of-the-box sparsification performance. Feather's effectiveness and adaptability is demonstrated using various architectures on the CIFAR dataset, while on ImageNet it achieves state-of-the-art Top-1 validation accuracy using the ResNet-50 architecture, surpassing existing methods, including more complex and computationally heavy ones, by a considerable margin. Code is publicly available at https://github.com/athglentis/feather .