Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
This addresses the problem of balancing latency and throughput for ML serving platforms, offering a novel approach to improve performance without harsh trade-offs.
The paper tackles the trade-off between latency and throughput in ML inference by introducing Apparate, a system that uses early exits to allow inputs to exit at intermediate layers, resulting in median response latency reductions of 40.5–91.5% for CV/NLP workloads and 22.6–77.9% for generative scenarios without compromising throughput or accuracy.
Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5--91.5% and 10.0--24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6--77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.