perf4sight: A toolflow to model CNN training performance on Edge GPUs
This addresses the need for efficient CNN retraining on resource-constrained edge devices, though it is incremental as it builds on existing modeling approaches.
The paper tackles the problem of predicting CNN training memory footprint and latency on edge GPUs to enable efficient network adaptation, achieving 95% accuracy for memory and 91% for latency on an NVIDIA Jetson TX2.
The increased memory and processing capabilities of today's edge devices create opportunities for greater edge intelligence. In the domain of vision, the ability to adapt a Convolutional Neural Network's (CNN) structure and parameters to the input data distribution leads to systems with lower memory footprint, latency and power consumption. However, due to the limited compute resources and memory budget on edge devices, it is necessary for the system to be able to predict the latency and memory footprint of the training process in order to identify favourable training configurations of the network topology and device combination for efficient network adaptation. This work proposes perf4sight, an automated methodology for developing accurate models that predict CNN training memory footprint and latency given a target device and network. This enables rapid identification of network topologies that can be retrained on the edge device with low resource consumption. With PyTorch as the framework and NVIDIA Jetson TX2 as the target device, the developed models predict training memory footprint and latency with 95% and 91% accuracy respectively for a wide range of networks, opening the path towards efficient network adaptation on edge GPUs.