Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding
This addresses the issue of slow inference speeds for users of large language models, offering a plug-and-play solution without retraining, though it is incremental as it builds on existing acceleration techniques.
The paper tackles the problem of high latency in large language models due to autoregressive processing by introducing Adaptive N-gram Parallel Decoding (ANPD), a lossless method that accelerates inference by generating multiple tokens simultaneously, achieving speed improvements up to 3.67x on models like LLaMA.
While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed by a verification phase, during which the original LLM assesses and confirms the proposed tokens. Consequently, ANPD preserves the integrity of the LLM's original output while enhancing processing speed. We further leverage a multi-level architecture for the N-gram module to enhance the precision of the initial draft, consequently reducing inference latency. ANPD eliminates the need for retraining or extra GPU memory, making it an efficient and plug-and-play enhancement. In our experiments, models such as LLaMA and its fine-tuned variants have shown speed improvements up to 3.67x, validating the effectiveness of our proposed ANPD.