Development of Quantized DNN Library for Exact Hardware Emulation
This addresses the need for precise hardware emulation in edge devices for developers and engineers, though it is incremental as it builds on existing quantization methods.
The researchers tackled the problem of inaccurate quantization emulation in deep neural networks by developing PyParch, a library that executes quantized DNNs with exact hardware behavior, enabling accurate overflow detection and precision estimation for complex models like YOLOv5, with emulation overheads of 5.6x slower for QNN and 42x slower with overflow detection compared to normal execution.
Quantization is used to speed up execution time and save power when runnning Deep neural networks (DNNs) on edge devices like AI chips. To investigate the effect of quantization, we need performing inference after quantizing the weights of DNN with 32-bit floating-point precision by a some bit width, and then quantizing them back to 32-bit floating-point precision. This is because the DNN library can only handle floating-point numbers. However, the accuracy of the emulation does not provide accurate precision. We need accurate precision to detect overflow in MAC operations or to verify the operation on edge de vices. We have developed PyParch, a DNN library that executes quantized DNNs (QNNs) with exactly the same be havior as hardware. In this paper, we describe a new proposal and implementation of PyParch. As a result of the evaluation, the accuracy of QNNs with arbitrary bit widths can be estimated for la rge and complex DNNs such as YOLOv5, and the overflow can be detected. We evaluated the overhead of the emulation time and found that it was 5.6 times slower for QNN and 42 times slower for QNN with overflow detection compared to the normal DNN execution time.