Dynamic Depth Decoding

“However, unlike , none of these techniques focus on the data movement cost due to speculation. They require access to output probability distributions and are incompatible with approaches like n-gram speculation. Also, they rely on aggressive drafting, assuming very low over-speculation penalties (1\u20132\% per unit increase in K), and must draft/verify at least one token to estimate benefits. Moreover, such schemes introduce CPU to GPU communication between drafter iterations on the GPU, to apply policy heuristics. Consequently, stopping criteria are used infrequently--e.g., DDD defers until the $5^{th}$ drafter iteration—making these methods too costly for MoEs.”