Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention
This work addresses the problem of efficient inference for Large Language Models, which is significant for natural language processing applications.
The authors tackled the problem of accelerating Large Language Models (LLMs) inference by proposing dynamic tree attention for multiple heads decoding, resulting in improved decoding efficiency while maintaining generation quality. Preliminary experiments demonstrated the potential for improvement of multiple head decoding in candidate generation.
Multiple heads decoding accelerates the inference of Large Language Models (LLMs) by predicting next several tokens simultaneously. It generates and verifies multiple candidate sequences in parallel via tree attention with a fixed structure. In this paper, we replace the fixed tree attention with dynamic tree attention on multiple head decoding, specifically in the context of MEDUSA. We propose a simple and low complexity strategy to generate candidates and construct the dynamic tree structure. Preliminary experiments show that the proposed method improves the decoding efficiency of multiple head decoding for LLMs while maintaining the generation quality. This result demonstrates the potential for improvement of multiple head decoding in candidate generation.