Learned Best-Effort LLM Serving
This addresses the challenge of cost-efficient LLM serving for applications requiring low latency, though it is incremental as it builds on existing serving systems with a learning-based optimization.
The paper tackles the problem of providing low-latency LLM service under fluctuating request patterns by introducing a best-effort serving system that uses deep reinforcement learning to adjust service quality based on task distribution and system load, resulting in over 10x higher client request rates and serving above 96% of peak performance 4.1x more often than static serving.
Many applications must provide low-latency LLM service to users or risk unacceptable user experience. However, over-provisioning resources to serve fluctuating request patterns is often prohibitively expensive. In this work, we present a best-effort serving system that employs deep reinforcement learning to adjust service quality based on the task distribution and system load. Our best-effort system can maintain availability with over 10x higher client request rates, serves above 96% of peak performance 4.1x more often, and serves above 98% of peak performance 2.3x more often than static serving on unpredictable workloads. Our learned router is robust to shifts in both the arrival and task distribution. Compared to static serving, learned best-effort serving allows for cost-efficient serving through increased hardware utility. Additionally, we argue that learned best-effort LLM serving is applicable in wide variety of settings and provides application developers great flexibility to meet their specific needs.