Slim Scheduler: A Runtime-Aware RL and Scheduler System for Efficient CNN Inference
This addresses the problem of optimizing CNN inference scheduling for heterogeneous hardware and fluctuating runtime conditions, representing an incremental improvement over static scheduling methods.
The paper tackles the problem of inefficient neural network scheduling for CNN inference by introducing Slim Scheduler, a hybrid framework that combines reinforcement learning with greedy schedulers to adapt to dynamic hardware conditions. It achieves up to 96.45% reduction in mean latency and 97.31% reduction in energy usage while maintaining accuracy trade-offs.
Most neural network scheduling research focuses on optimizing static, end-to-end models of fixed width, overlooking dynamic approaches that adapt to heterogeneous hardware and fluctuating runtime conditions. We present Slim Scheduler, a hybrid scheduling framework that integrates a Proximal Policy Optimization (PPO) reinforcement learning policy with algorithmic, greedy schedulers to coordinate distributed inference for slimmable models. Each server runs a local greedy scheduler that batches compatible requests and manages instance scaling based on VRAM and utilization constraints, while the PPO router learns global routing policies for device selection, width ratio, and batch configuration. This hierarchical design reduces search space complexity, mitigates overfitting to specific hardware, and balances efficiency and throughput. Compared to a purely randomized task distribution baseline, Slim Scheduler can achieve various accuracy and latency trade-offs such as: A 96.45% reduction in mean latency and a 97.31% reduction in energy usage dropping accuracy to the slimmest model available (70.3%). It can then accomplish an overall reduction in average latency plus energy consumption with an increase in accuracy at the cost of higher standard deviations of said latency and energy, effecting overall task throughput.