HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network
This addresses the problem of slow and unreliable LLM inference at the edge for applications requiring low latency and privacy, representing a strong incremental improvement over existing distributed methods.
The paper tackles the challenge of distributed LLM inference in unreliable edge networks by proposing HALO, a framework that uses semantic-aware allocation to reduce synchronization delays, achieving a 3.41x speedup for LLaMA models on a Raspberry Pi cluster while maintaining performance comparable to optimal conditions.
The deployment of large language models' (LLMs) inference at the edge can facilitate prompt service responsiveness while protecting user privacy. However, it is critically challenged by the resource constraints of a single edge node. Distributed inference has emerged to aggregate and leverage computational resources across multiple devices. Yet, existing methods typically require strict synchronization, which is often infeasible due to the unreliable network conditions. In this paper, we propose HALO, a novel framework that can boost the distributed LLM inference in lossy edge network. The core idea is to enable a relaxed yet effective synchronization by strategically allocating less critical neuron groups to unstable devices, thus avoiding the excessive waiting time incurred by delayed packets. HALO introduces three key mechanisms: (1) a semantic-aware predictor to assess the significance of neuron groups prior to activation. (2) a parallel execution scheme of neuron group loading during the model inference. (3) a load-balancing scheduler that efficiently orchestrates multiple devices with heterogeneous resources. Experimental results from a Raspberry Pi cluster demonstrate that HALO achieves a 3.41x end-to-end speedup for LLaMA-series LLMs under unreliable network conditions. It maintains performance comparable to optimal conditions and significantly outperforms the state-of-the-art in various scenarios.