HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization
This addresses the need for resource-efficient machine learning applications, though it appears incremental as it builds on existing sparse learning methods.
The paper tackles the problem of compressing dense neural networks into sparse ones for resource efficiency by proposing Adaptive Regularized Training (ART) with a HyperSparse loss, achieving notable performance gains in high sparsity regimes up to 99.8% on datasets like CIFAR and TinyImageNet.
Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model knowledge into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8 percent model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.