SpecServe: Efficient and SLO-Aware Large Language Model Serving with Adaptive Speculative Decoding
This addresses performance variability and SLO violations in LLM serving, which is crucial for real-time applications, though it is incremental as it builds on existing speculative decoding techniques.
The paper tackles the problem of low inference latency and Service Level Objective (SLO) violations in Large Language Model (LLM) serving under dynamic workloads by introducing SpecServe, a system that dynamically adjusts speculative decoding strategies, resulting in speedups of 1.14× to 14.3× over state-of-the-art systems.
Large Language Model (LLM) services often face challenges in achieving low inference latency and meeting Service Level Objectives (SLOs) under dynamic request patterns. Speculative decoding, which exploits lightweight models for drafting and LLMs for verification, has emerged as a compelling technique to accelerate LLM inference. However, existing speculative decoding solutions often fail to adapt to varying workloads and system environments, resulting in performance variability and SLO violations. In this paper, we introduce SpecServe, an efficient LLM inference system that dynamically adjusts speculative strategies according to real-time request loads and system configurations. SpecServe proposes a theoretical model to understand and predict the efficiency of speculative decoding across diverse scenarios. Additionally, it implements intelligent drafting and verification algorithms to guarantee optimal performance while achieving high SLO attainment. Experimental results on real-world LLM traces demonstrate that SpecServe consistently meets SLOs and achieves substantial performance improvements, yielding 1.14$\times$-14.3$\times$ speedups over state-of-the-art speculative inference systems.