Dynamic Network Adaptation at Inference
This addresses the problem of real-time inference workloads for ML serving systems, offering a novel method to adapt models dynamically, though it is incremental in optimizing existing trade-offs.
The paper tackles the challenge of meeting strict Service Level Objectives (SLOs) like latency and accuracy in machine learning inference by proposing SLO-Aware Neural Networks that dynamically drop nodes per query, achieving average speedups of 1.3-56.7x with less than 0.3% accuracy loss.
Machine learning (ML) inference is a real-time workload that must comply with strict Service Level Objectives (SLOs), including latency and accuracy targets. Unfortunately, ensuring that SLOs are not violated in inference-serving systems is challenging due to inherent model accuracy-latency tradeoffs, SLO diversity across and within application domains, evolution of SLOs over time, unpredictable query patterns, and co-location interference. In this paper, we observe that neural networks exhibit high degrees of per-input activation sparsity during inference. . Thus, we propose SLO-Aware Neural Networks which dynamically drop out nodes per-inference query, thereby tuning the amount of computation performed, according to specified SLO optimization targets and machine utilization. SLO-Aware Neural Networks achieve average speedups of $1.3-56.7\times$ with little to no accuracy loss (less than 0.3%). When accuracy constrained, SLO-Aware Neural Networks are able to serve a range of accuracy targets at low latency with the same trained model. When latency constrained, SLO-Aware Neural Networks can proactively alleviate latency degradation from co-location interference while maintaining high accuracy to meet latency constraints.