Towards Fair and Firm Real-Time Scheduling in DNN Multi-Tenant Multi-Accelerator Systems via Reinforcement Learning
It addresses QoS management for cloud service providers and tenants, but appears incremental as it builds on existing scheduling methods with a reinforcement learning twist.
This paper tackles the problem of managing Quality of Service (QoS) in multi-tenant, multi-accelerator cloud systems for DNNs by introducing a reinforcement learning-based online scheduling algorithm to guarantee tenant-specific QoS levels, such as deadline hit rates, under real-time constraints.
This paper addresses the critical challenge of managing Quality of Service (QoS) in cloud services, focusing on the nuances of individual tenant expectations and varying Service Level Indicators (SLIs). It introduces a novel approach utilizing Deep Reinforcement Learning for tenant-specific QoS management in multi-tenant, multi-accelerator cloud environments. The chosen SLI, deadline hit rate, allows clients to tailor QoS for each service request. A novel online scheduling algorithm for Deep Neural Networks in multi-accelerator systems is proposed, with a focus on guaranteeing tenant-wise, model-specific QoS levels while considering real-time constraints.