Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
This addresses the problem of slow LLM inference for users by reducing latency through parallelization, representing a novel method rather than an incremental improvement.
The paper tackles the high latency of autoregressive decoding in large language models by introducing Lookahead decoding, an exact parallel algorithm that accelerates inference without auxiliary models, achieving speedups of up to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code completion tasks.
Autoregressive decoding of large language models (LLMs) is memory bandwidth bounded, resulting in high latency and significant wastes of the parallel processing power of modern accelerators. Existing methods for accelerating LLM decoding often require a draft model (e.g., speculative decoding), which is nontrivial to obtain and unable to generalize. In this paper, we introduce Lookahead decoding, an exact, parallel decoding algorithm that accelerates LLM decoding without needing auxiliary models or data stores. It allows trading per-step log(FLOPs) to reduce the number of total decoding steps, is more parallelizable on single or multiple modern accelerators, and is compatible with concurrent memory-efficient attention (e.g., FlashAttention). Our implementation of Lookahead decoding can speed up autoregressive decoding by up to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code completion tasks. Our code is avialable at https://github.com/hao-ai-lab/LookaheadDecoding