Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
This provides a practical method for model compression in resource-constrained environments, though it is incremental as it builds on existing training techniques.
The paper tackles the problem of reducing large deep neural network models for edge devices by showing that training with ReLU activations, L2 regularization, and the Adam optimizer automatically induces group sparsity in weights, enabling efficient model reduction without extra regularization.
In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation. However, large DNN models are needed to achieve state-of-the-art performance, exceeding the capabilities of edge devices. Model reduction is thus needed for practical use. In this paper, we point out that deep learning automatically induces group sparsity of weights, in which all weights connected to an output channel (node) are zero, when training DNNs under the following three conditions: (1) rectified-linear-unit (ReLU) activations, (2) an $L_2$-regularized objective function, and (3) the Adam optimizer. Next, we analyze this behavior both theoretically and experimentally, and propose a simple model reduction method: eliminate the zero weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets, we demonstrate the sparsity with various training setups. Finally, we show that our method can efficiently reduce the model size and performs well relative to methods that use a sparsity-inducing regularizer.