Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms
This work addresses runtime performance scaling for DNNs on mobile and embedded platforms, offering a novel method that avoids costly retraining, though it is incremental in building upon existing NAS models.
The paper tackles the problem of efficiently running DNNs on heterogeneous embedded platforms with varying runtime resources and performance requirements by proposing Dynamic-OFA, which enables runtime switching between pre-sampled sub-networks from a static Once-for-all network without retraining. Experimental results on ImageNet with a Jetson Xavier NX show speedups of up to 3.5x on CPU and 2.4x on GPU for similar accuracy, or accuracy gains of 3.8% on CPU and 5.1% on GPU at similar latency.
Mobile and embedded platforms are increasingly required to efficiently execute computationally demanding DNNs across heterogeneous processing elements. At runtime, the available hardware resources to DNNs can vary considerably due to other concurrently running applications. The performance requirements of the applications could also change under different scenarios. To achieve the desired performance, dynamic DNNs have been proposed in which the number of channels/layers can be scaled in real time to meet different requirements under varying resource constraints. However, the training process of such dynamic DNNs can be costly, since platform-aware models of different deployment scenarios must be retrained to become dynamic. This paper proposes Dynamic-OFA, a novel dynamic DNN approach for state-of-the-art platform-aware NAS models (i.e. Once-for-all network (OFA)). Dynamic-OFA pre-samples a family of sub-networks from a static OFA backbone model, and contains a runtime manager to choose different sub-networks under different runtime environments. As such, Dynamic-OFA does not need the traditional dynamic DNN training pipeline. Compared to the state-of-the-art, our experimental results using ImageNet on a Jetson Xavier NX show that the approach is up to 3.5x (CPU), 2.4x (GPU) faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU) higher accuracy at similar latency.