PLD+: Accelerating LLM inference by leveraging Language Model Artifacts
This work addresses the practical deployment challenge of speculative decoding for LLM inference by providing a tuning-free solution that reduces latency without additional compute, benefiting users in input-guided tasks like code editing and summarization.
The paper tackles the latency problem in autoregressive LLM inference for input-guided tasks by introducing PLD+, a tuning-free method that leverages language model artifacts and input-output overlap to accelerate inference, achieving up to 2.31x average speedup and outperforming state-of-the-art tuning-dependent approaches on four tasks.
To reduce the latency associated with autoretrogressive LLM inference, speculative decoding has emerged as a novel decoding paradigm, where future tokens are drafted and verified in parallel. However, the practical deployment of speculative decoding is hindered by its requirements for additional computational resources and fine-tuning, which limits its out-of-the-box usability. To address these challenges, we present PLD+, a suite of novel algorithms developed to accelerate the inference process of LLMs, particularly for input-guided tasks. These tasks, which include code editing, text editing, summarization, etc., often feature outputs with substantial overlap with their inputs-an attribute PLD+ is designed to exploit. PLD+ also leverages the artifacts (attention and hidden states) generated during inference to accelerate inference speed. We test our approach on five input-guided tasks and through extensive experiments we find that PLD+ outperforms all tuning-free approaches. In the greedy setting, it even outperforms the state-of-the-art tuning-dependent approach EAGLE on four of the tasks. (by a margin of upto 2.31 in terms of avg. speedup). Our approach is tuning free, does not require any additional compute and can easily be used for accelerating inference of any LLM.