Speculative decoding
Dynamic Depth Decoding
Dynamic Depth Decoding: Faster Speculative Decoding for LLMs
Superseded baseline#101 of 151 most-superseded · first seen Aug 30, 2024
Cited as a baseline — critiqued by newer work, not yet beaten on a benchmark here
1 papers critique it · 0 beat it on benchmarks
What papers say
Verbatim critique sentences, each from a paper that cites Dynamic Depth Decoding as a baseline.
However, unlike , none of these techniques focus on the data movement cost due to speculation. They require access to output probability distributions and are incompatible with approaches like n-gram speculation. Also, they rely on aggressive drafting, assuming very low over-speculation penalties (1\u20132\% per unit increase in K), and must draft/verify at least one token to estimate benefits. Moreover, such schemes introduce CPU to GPU communication between drafter iterations on the GPU, to apply policy heuristics. Consequently, stopping criteria are used infrequently--e.g., DDD defers until the $5^{th}$ drafter iteration—making these methods too costly for MoEs.
What to use instead
Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.
- Nov 3, 2025