Scaling Up Exact Neural Network Compression by ReLU Stability
This work addresses the computational bottleneck in exact neural network compression for practitioners, though it is incremental as it builds on existing stability-based methods.
The paper tackles the problem of compressing neural networks exactly by identifying stable ReLU neurons more efficiently, achieving a median speedup of 183 times over the state-of-the-art on CIFAR-10 and enabling compression of up to 56% of connections without accuracy loss.
We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 56% of the connections on the CIFAR-10 dataset. The code is available at the following link, https://github.com/yuxwind/ExactCompression.