SpecInfer

“SpecInfer incurs additional tree construction and tree verification overhead, while draft generation and target verification remain serialized.”

“Since the original speculative decoding algorithm generates a single draft sequence, it is not competitive when drafting thousands of tokens.”

“While these methods aim to improve inference speed by increasing the token acceptance rate, they do not guarantee full recovery of the target distribution.”

“The 16%–66% additional speedup over SpecInfer* reveals the inefficiency of verifying large batches of draft tokens in complex request patterns.”

“For instance, miao2023specinfer simply samples k independent sequences as predicted tokens, while chen2024sequoia fix the tree structure learned from training distribution for every test query. However, fixed patterns usually struggle to generalize to diverse query distributions, resulting in a relatively low acceptance rate as tree size grows.”

“miao2023specinfer employ multiple draft models to generate tokens and merge them using tree attention, while spector2023accelerating utilize a small draft model to process each level of the tree in batches. In contrast, our method directly uses the top predicted tokens from each of heads to create a static sparse tree without autoregression or adjusting the tree structure.”

“SpecInfer suffers from the latency of running another parametric model”

“For multi-request inference, the speculative window is constrained to a single step, implying that only small and potentially inaccurate draft models are deployable.”

“we observe that existing token tree construction algorithms perform well for small token trees but are sub-optimal for large tree sizes. For example, SpecInfer constructs a token tree using $k$ independent sequences, a topology that is bounded by the expected number of tokens it can accept, regardless of the tree size”

“necessitate customized training and the reliance on an auxiliary draft model also introduces memory overheads”