MiCo: End-to-End Mixed Precision Neural Network Co-Exploration Framework for Edge AI
This work addresses the need for flexible and efficient optimization of quantized neural networks for edge AI applications, representing an incremental improvement over existing methods.
The paper tackles the challenge of efficiently exploring and deploying layer-wise mixed-precision quantization schemes for neural networks on edge devices, resulting in a framework that achieves end-to-end speedup with minimal accuracy drops.
Quantized Neural Networks (QNN) with extremely low-bitwidth data have proven promising in efficient storage and computation on edge devices. To further reduce the accuracy drop while increasing speedup, layer-wise mixed-precision quantization (MPQ) becomes a popular solution. However, existing algorithms for exploring MPQ schemes are limited in flexibility and efficiency. Comprehending the complex impacts of different MPQ schemes on post-training quantization and quantization-aware training results is a challenge for conventional methods. Furthermore, an end-to-end framework for the optimization and deployment of MPQ models is missing in existing work. In this paper, we propose the MiCo framework, a holistic MPQ exploration and deployment framework for edge AI applications. The framework adopts a novel optimization algorithm to search for optimal quantization schemes with the highest accuracies while meeting latency constraints. Hardware-aware latency models are built for different hardware targets to enable fast explorations. After the exploration, the framework enables direct deployment from PyTorch MPQ models to bare-metal C codes, leading to end-to-end speedup with minimal accuracy drops.