Exploring and Improving Drafts in Blockwise Parallel Decoding
This work addresses inference speed bottlenecks for users of large language models, but it is incremental as it builds on existing blockwise parallel decoding methods.
The paper tackled the problem of slow inference speeds in autoregressive language models by improving blockwise parallel decoding, resulting in a 5-21% increase in block efficiency across diverse datasets.
Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation. Blockwise parallel decoding (BPD) was proposed by Stern et al. as a method to improve inference speed of language models by simultaneously predicting multiple future tokens, termed block drafts, which are subsequently verified and conditionally accepted by the autoregressive model. This paper contributes to the understanding and improvement of block drafts in two ways. First, we analyze the token distributions produced by multiple prediction heads. Secondly, we leverage this analysis to develop algorithms to improve BPD inference speed by refining the block drafts using n-gram and neural language models. Experiments demonstrate that refined block drafts yield a +5-21% increase in block efficiency (i.e., the number of accepted tokens from the block draft) across diverse datasets.