Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution
This work addresses the problem of slow and inefficient text generation on mobile devices for applications like intelligent assistants and UI agents, representing an incremental improvement through optimized hardware scheduling and speculative decoding.
The paper tackles the high latency and low hardware utilization in on-device large language model generation by introducing sd.npu, a mobile inference framework that integrates speculative decoding with dynamic hardware scheduling, achieving up to 3.8x faster generation and 4.7x better energy efficiency compared to existing solutions.
Enhancing on-device large language models (LLMs) with contextual information from local data enables personalized and task-aware generation, powering use cases such as intelligent assistants and UI agents. While recent developments in neural processors have substantially improved the efficiency of prefill on mobile devices, the token-by-token generation process still suffers from high latency and limited hardware utilization due to its inherently memory-bound characteristics. This work presents sd.npu, a mobile inference framework that integrates speculative decoding with dynamic hardware scheduling to accelerate context-aware text generation on mobile devices. The framework introduces three synergistic components: (1) adaptive execution scheduling, which dynamically balances compute graphs between prefill and decoding phases; (2) context-aligned drafting, which improves speculative efficiency through lightweight online calibration to current tasks; and (3) hardware-efficient draft extension, which reuses and expands intermediate sequences to improve processing parallelism and reduce verification cost. Experiments on multiple smartphones and representative workloads show consistent improvements of up to 3.8x in generation speed and 4.7x in energy efficiency compared with existing mobile inference solutions. Component-level analysis further validates the contribution of each optimization.