Medusa

“the Medusa head consists of only a single MLP layer that takes input solely from the final hidden states. Each layer independently speculates on a word at a specified position beyond the next, disregarding the sequential dependencies from previously predicted tokens, which often results in decreased accuracy”

“Medusa also hurts MoE performance, as it increases the in-flight tokens by 50-100x and would activate all experts every iteration, for a cost increase of 4x-8x depending on the MoE sparsity, while the ETR increase rarely justifies the cost.”

“Methods like Medusa relax acceptance conditions under non-greedy settings, which do not guarantee lossless acceleration.”

“Tree-attention frameworks—SpecInfer~miao2024specinfer, Medusa~cai2024medusa, and Eagle~li2024eagle, fan2026flatter—expand many branches, quickly exhausting memory.”

“However, Medusa build the token tree directly based on the probability of draft model, instead of a mapping between sampling of draft model and sampling of target model.”

“Although Medusa eliminates the overhead of maintaining an independent draft model, its non-autoregressive MLP heads struggle to capture long-range dependencies.”

“Medusa applies lightweight decoding heads to predict multiple subsequent tokens on the top-layer features of the target model but delivers limited accuracy.”

“All draft heads to date make predictions only as a function of the base model's hidden states from previously verified tokens, making them unaware of earlier tokens in the current candidate continuation. Because of the strong statistical dependencies between neighboring tokens in language, this sequential independence limits the prediction accuracy of existing draft head architectures.”

“it relies solely on hidden states from previously verified tokens, making it blind to earlier unverified predictions within the current draft round.”

“Once alternatives from different depths are combined into a draft tree, they form a large combinatorial space in which many paths are not coherent continuations, and the verifier wastes budget on them.”

“Prevailing methods, medusa, li2024eagle, vicuna68m use small drafters simply trained on datasets such as ShareGPT sharegpt which is often used for instruction tuning of LLMs to learn a pattern of target LLM's language modeling. However, our investigations reveal that such approaches are insufficient for multilingual translation.”

“methods such as Medusa~cai2024medusa eliminate the dependency between heads, thereby accelerating the generation of drafts. However, these methods primarily focus on modeling Syntactic Coherence while neglecting Semantic Coherence.”