Deep Compression for PyTorch Model Deployment on Microcontrollers
This work addresses the challenge of efficient model deployment on low-cost embedded systems for applications like IoT, but it is incremental as it builds on prior work.
The paper tackles the problem of deploying neural networks on microcontrollers with limited memory and compute by applying Deep Compression to PyTorch models, resulting in a 12.45x reduction in memory footprint and a 2.57x boost in inference speed for the LeNet-5 model.
Neural network deployment on low-cost embedded systems, hence on microcontrollers (MCUs), has recently been attracting more attention than ever. Since MCUs have limited memory capacity as well as limited compute-speed, it is critical that we employ model compression, which reduces both memory and compute-speed requirements. In this paper, we add model compression, specifically Deep Compression, and further optimize Unlu's earlier work on arXiv, which efficiently deploys PyTorch models on MCUs. First, we prune the weights in convolutional and fully connected layers. Secondly, the remaining weights and activations are quantized to 8-bit integers from 32-bit floating-point. Finally, forward pass functions are compressed using special data structures for sparse matrices, which store only nonzero weights (without impacting performance and accuracy). In the case of the LeNet-5 model, the memory footprint was reduced by 12.45x, and the inference speed was boosted by 2.57x.