Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU
This addresses the challenge of efficient multi-DNN inference for applications like autonomous driving and augmented reality on edge devices, representing an incremental improvement in task coordination for specific hardware constraints.
The paper tackles the problem of coordinating multiple deep neural networks (DNNs) with varying real-time requirements on resource-limited edge GPUs, proposing Miriam, a contention-aware task coordination framework that increases system throughput by 92% while keeping latency overhead for critical tasks under 10% compared to baselines.
Many applications such as autonomous driving and augmented reality, require the concurrent running of multiple deep neural networks (DNN) that poses different levels of real-time performance requirements. However, coordinating multiple DNN tasks with varying levels of criticality on edge GPUs remains an area of limited study. Unlike server-level GPUs, edge GPUs are resource-limited and lack hardware-level resource management mechanisms for avoiding resource contention. Therefore, we propose Miriam, a contention-aware task coordination framework for multi-DNN inference on edge GPU. Miriam consolidates two main components, an elastic-kernel generator, and a runtime dynamic kernel coordinator, to support mixed critical DNN inference. To evaluate Miriam, we build a new DNN inference benchmark based on CUDA with diverse representative DNN workloads. Experiments on two edge GPU platforms show that Miriam can increase system throughput by 92% while only incurring less than 10\% latency overhead for critical tasks, compared to state of art baselines.