Collaborative Large Language Model Inference via Resource-Aware Parallel Speculative Decoding
This work addresses the need for scalable and low-latency LLM services in resource-constrained mobile edge computing settings, representing an incremental improvement over existing speculative decoding methods.
The paper tackles the problem of communication overhead and delays in speculative decoding for large language model inference in mobile edge computing by proposing a unified framework that jointly optimizes user association and resource allocation. It achieves up to 28.0% and an average of 23.7% reduction in end-to-end latency without compromising accuracy.
The growing demand for on-device large language model (LLM) inference highlights the need for efficient mobile edge computing (MEC) solutions, especially in resource-constrained settings. Speculative decoding offers a promising solution by partitioning token generation between a lightweight draft model on mobile devices and a powerful target model on edge servers, but suffers from communication overhead and asynchronous delays. This paper is the first to propose a unified framework that jointly optimizes user association and resource allocation (UARA) to support efficient parallel speculative decoding. We solve the UARA problem using a multi-agent deep reinforcement learning algorithm. To evaluate our approach under realistic conditions, we conduct experiments using the Sionna simulator. Results show that our method achieves up to 28.0% and an average of 23.7% reduction in end-to-end latency without compromising inference accuracy, enabling scalable and low-latency LLM services in MEC systems.