Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
This work addresses the challenge of slow distributed LLM inference for users in decentralized settings by providing a system-level optimization that converts network stalls into throughput without requiring model retraining or architectural changes.
The paper tackles the problem of network latency dominating compute in decentralized large language model inference by introducing Decentralized Speculative Decoding (DSD), a framework that turns communication delay into useful computation through parallel verification of candidate tokens, achieving up to 2.56x speedup on benchmarks like HumanEval and GSM8K while preserving accuracy.
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens that are later verified by a stronger target model. While effective in centralized systems, its behavior in decentralized settings, where network latency often dominates compute, remains under-characterized. We present Decentralized Speculative Decoding (DSD), a plug-and-play framework for decentralized inference that turns communication delay into useful computation by verifying multiple candidate tokens in parallel across distributed nodes. We further introduce an adaptive speculative verification strategy that adjusts acceptance thresholds by token-level semantic importance, delivering an additional 15% to 20% end-to-end speedup without retraining. In theory, DSD reduces cross-node communication cost by approximately (N-1)t1(k-1)/k, where t1 is per-link latency and k is the average number of tokens accepted per round. In practice, DSD achieves up to 2.56x speedup on HumanEval and 2.59x on GSM8K, surpassing the Eagle3 baseline while preserving accuracy. These results show that adapting speculative decoding for decentralized execution provides a system-level optimization that converts network stalls into throughput, enabling faster distributed LLM inference with no model retraining or architectural changes.