On-Device Neural Net Inference with Mobile GPUs
This work addresses the problem of efficient and accessible on-device inference for mobile app developers and users, offering a solution that works on virtually all phones, though it is incremental by building on existing hardware acceleration trends.
The paper tackles the challenge of on-device neural network inference on mobile phones by leveraging the ubiquitous mobile GPU, achieving real-time performance for both Android and iOS devices, and integrates this engine into TensorFlow Lite.
On-device inference of machine learning models for mobile phones is desirable due to its lower latency and increased privacy. Running such a compute-intensive task solely on the mobile CPU, however, can be difficult due to limited computing power, thermal constraints, and energy consumption. App developers and researchers have begun exploiting hardware accelerators to overcome these challenges. Recently, device manufacturers are adding neural processing units into high-end phones for on-device inference, but these account for only a small fraction of hand-held devices. In this paper, we present how we leverage the mobile GPU, a ubiquitous hardware accelerator on virtually every phone, to run inference of deep neural networks in real-time for both Android and iOS devices. By describing our architecture, we also discuss how to design networks that are mobile GPU-friendly. Our state-of-the-art mobile GPU inference engine is integrated into the open-source project TensorFlow Lite and publicly available at https://tensorflow.org/lite.