Towards Hardware-Specific Automatic Compression of Neural Networks
This work addresses the need for efficient model deployment on embedded or mobile devices by providing hardware-specific compression, though it is incremental as it builds on existing pruning and quantization methods.
The authors tackled the problem of automatically compressing neural networks for specific hardware by proposing Galen, a reinforcement learning framework that optimizes pruning and quantization policies using measured inference latency, achieving a compression of ResNet18 on CIFAR-10 to 20% of original latency without significant accuracy loss.
Compressing neural network architectures is important to allow the deployment of models to embedded or mobile devices, and pruning and quantization are the major approaches to compress neural networks nowadays. Both methods benefit when compression parameters are selected specifically for each layer. Finding good combinations of compression parameters, so-called compression policies, is hard as the problem spans an exponentially large search space. Effective compression policies consider the influence of the specific hardware architecture on the used compression methods. We propose an algorithmic framework called Galen to search such policies using reinforcement learning utilizing pruning and quantization, thus providing automatic compression for neural networks. Contrary to other approaches we use inference latency measured on the target hardware device as an optimization goal. With that, the framework supports the compression of models specific to a given hardware target. We validate our approach using three different reinforcement learning agents for pruning, quantization and joint pruning and quantization. Besides proving the functionality of our approach we were able to compress a ResNet18 for CIFAR-10, on an embedded ARM processor, to 20% of the original inference latency without significant loss of accuracy. Moreover, we can demonstrate that a joint search and compression using pruning and quantization is superior to an individual search for policies using a single compression method.