Lookahead

“This would significantly increase the in-flight tokens (by $G$) for MoEs, and our evaluations reveal that even vanilla n-gram decoding ($G=1$) incurs high costs.”

“some speculative decoding approaches, such as Lookahead, rely solely on N-gram or retrieval-based heuristics for drafting. While such methods incur negligible drafting latency, their limited predictive accuracy typically leads to very low average acceptance length $$, resulting in modest end-to-end speedups.”

“These methods refine multiple tokens simultaneously using the model's internal attention mechanism, eliminating the need for explicit draft weights, though often yielding shorter acceptance lengths compared to model-based drafters.”

“this approach makes each decode step significantly more computationally expensive”

“While these methods substantially improve proposal quality, their objectives are typically defined at the token or local distribution level, leaving the window-level and prefix-sensitive nature of speculative verification less explicitly optimized.”

“Prompt- and retrieval-based approaches (PLD, Lookahead, CLLMs) improve draft quality but degrade with scarce context”

“However, these methods exhibit lower accuracy and greater resource use compared to our approach. They demand more memory and GPU processing power, posing challenges in resource-scarce settings.”

“However, due to its lower efficiency in generating draft tokens compared to Medusa, its end-to-end speedup ratio is slightly lower than that of Medusa”