26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone
This work addresses the challenge of real-time DNN inference on smartphones, which is crucial for applications like video stream processing, though it appears incremental as it builds on existing optimization techniques.
The paper tackles the problem of efficiently executing deep neural networks on mobile devices by introducing CADNN, a framework that uses model compression and architecture-aware optimizations, achieving a 26ms inference time for ResNet-50 and outperforming state-of-the-art frameworks like TensorFlow Lite and TVM.
With the rapid emergence of a spectrum of high-end mobile devices, many applications that required desktop-level computation capability formerly can now run on these devices without any problem. However, without a careful optimization, executing Deep Neural Networks (a key building block of the real-time video stream processing that is the foundation of many popular applications) is still challenging, specifically, if an extremely low latency or high accuracy inference is needed. This work presents CADNN, a programming framework to efficiently execute DNN on mobile devices with the help of advanced model compression (sparsity) and a set of thorough architecture-aware optimization. The evaluation result demonstrates that CADNN outperforms all the state-of-the-art dense DNN execution frameworks like TensorFlow Lite and TVM.