C2T: A Classifier-Based Tree Construction Method in Speculative Decoding
This addresses inference latency and computational costs in LLMs, representing an incremental improvement over existing speculative decoding methods.
The paper tackles inefficiencies in speculative decoding for LLMs by proposing C2T, a method that uses a lightweight classifier to dynamically generate and prune token trees, reducing candidate tokens by 25% while maintaining acceptance length compared to SOTA methods like EAGLE-2.
The growing scale of Large Language Models (LLMs) has exacerbated inference latency and computational costs. Speculative decoding methods, which aim to mitigate these issues, often face inefficiencies in the construction of token trees and the verification of candidate tokens. Existing strategies, including chain mode, static tree, and dynamic tree approaches, have limitations in accurately preparing candidate token trees for verification. We propose a novel method named C2T that adopts a lightweight classifier to generate and prune token trees dynamically. Our classifier considers additional feature variables beyond the commonly used joint probability to predict the confidence score for each draft token to determine whether it is the candidate token for verification. This method outperforms state-of-the-art (SOTA) methods such as EAGLE-2 on multiple benchmarks, by reducing the total number of candidate tokens by 25% while maintaining or even improving the acceptance length.