EAGLE

“there exists a conflict between the feature-level and the logit-level losses. The feature-level loss is introduced to facilitate knowledge distillation. However, to the best of our knowledge, there is currently no research indicating that the knowledge of an LLM can be distilled into a single transformer decoder layer. We believe that performing strict knowledge distillation between the target LLM and a lightweight draft model is unrealistic.”

“these methods still necessitate a computationally expensive projection over the full vocabulary at the draft head's final layer, which persists as a major latency source”

“the top-$k$ tokens within the same layer of the draft tree remain coupled, which limits the diversity and potential of the predictions.”

“The shape of EAGLE's draft tree is fixed, with the drafting phase filling in the corresponding positions. EAGLE-2 aims to improve this by introducing a dynamically adjustable draft tree.”

“Tree-attention frameworks—SpecInfer~miao2024specinfer, Medusa~cai2024medusa, and Eagle~li2024eagle, fan2026flatter—expand many branches, quickly exhausting memory.”

“its structure is static with no dependence on the draft model output”

“Due to the sampling results at the token layer being hidden, feature-level autoregression introduces uncertainty.”

“For MoE models, however, this assumption breaks down; we analyze this problem in sec:method:problem.”

“only EAGLE~li2024eagle presents results for batch sizes $ 4$ but doesn't discuss larger batch sizes.”

“Prevailing methods, medusa, li2024eagle, vicuna68m use small drafters simply trained on datasets such as ShareGPT sharegpt which is often used for instruction tuning of LLMs to learn a pattern of target LLM's language modeling. However, our investigations reveal that such approaches are insufficient for multilingual translation.”

“target-dependent methods such as the EAGLE series require separate training for each individual model”

“methods like Eagle~li2024eagle, which use an autoregressive approach, are more suitable for modeling Semantic Coherence. Using such methods to model Syntactic Coherence introduces unnecessary computational overhead.”