Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
This work addresses computational efficiency for deep learning practitioners, but it is incremental as it builds on known sparsity properties.
The paper tackled the problem of accelerating convolutional neural networks by exploiting the sparsity of ReLU outputs, achieving speedup improvements on CPUs when sparsity is at least 0.9.
Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks. An ReLU converts negative values to zeros, and does not change positive values, which leads to a high sparsity of neurons. In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures. And then we use the sparsity property of ReLUs to accelerate the calculation of convolution by skipping calculations of zero-valued neurons. The proposed sparse convolution algorithm achieves some speedup improvements on CPUs compared to the traditional matrix-matrix multiplication algorithm for convolution when the sparsity is not less than 0.9.