OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workload
This addresses the problem of low throughput in multi-DNN applications on heterogeneous embedded devices for developers and system designers, representing a strong specific gain rather than a foundational advancement.
The paper tackles the challenge of efficiently distributing multi-DNN workloads across heterogeneous embedded accelerators, achieving an average throughput boost of 4.6x compared to state-of-the-art methods.
Modern Deep Neural Networks (DNNs) exhibit profound efficiency and accuracy properties. This has introduced application workloads that comprise of multiple DNN applications, raising new challenges regarding workload distribution. Equipped with a diverse set of accelerators, newer embedded system present architectural heterogeneity, which current run-time controllers are unable to fully utilize. To enable high throughput in multi-DNN workloads, such a controller is ought to explore hundreds of thousands of possible solutions to exploit the underlying heterogeneity. In this paper, we propose OmniBoost, a lightweight and extensible multi-DNN manager for heterogeneous embedded devices. We leverage stochastic space exploration and we combine it with a highly accurate performance estimator to observe a x4.6 average throughput boost compared to other state-of-the-art methods. The evaluation was performed on the HiKey970 development board.