DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving
This addresses the need for agile and scalable LLM serving in heterogeneous edge-cloud settings, representing an incremental improvement over single-node speculative decoding.
The paper tackles the problem of high decoding latency and limited scalability in large language model inference across edge-cloud environments by proposing DSD, a distributed speculative decoding framework, which achieves up to 1.1x speedup and 9.7% higher throughput over existing baselines.
Large language model (LLM) inference often suffers from high decoding latency and limited scalability across heterogeneous edge-cloud environments. Existing speculative decoding (SD) techniques accelerate token generation but remain confined to single-node execution. We propose DSD, a distributed speculative decoding framework that extends SD to multi-device deployments through coordinated draft-target execution. Given the lack of prior work on simulating this paradigm, we first introduce DSD-Sim, a discrete-event simulator that captures network, batching, and scheduling dynamics. Building on insights from DSD-Sim, we further design an Adaptive Window Control (AWC) policy that dynamically adjusts speculation window size to optimize throughput. Experiments across diverse workloads show that DSD achieves up to 1.1x speedup and 9.7% higher throughput over existing SD baselines, enabling agile and scalable LLM serving across edge and cloud.