Flatter Tokens are More Valuable for Speculative Draft Model Training
This work addresses the training efficiency problem for Speculative Decoding in LLM inference, offering a data-centric solution that is incremental but provides substantial practical gains.
The paper tackles the inefficiency of training draft models for Speculative Decoding by identifying that tokens with flatter predictive distributions are more valuable, and proposes a data filtering method (SFDD) that achieves over 2x training speedup with 50% of the data while maintaining inference speedup within 4% of the baseline.
Speculative Decoding (SD) is a key technique for accelerating Large Language Model (LLM) inference, but it typically requires training a draft model on a large dataset. We approach this problem from a data-centric perspective, finding that not all training samples contribute equally to the SD acceptance rate. Specifically, our theoretical analysis and empirical validation reveals that tokens inducing flatter predictive distributions from the target model are more valuable than those yielding sharply peaked distributions. Based on this insight, we propose flatness, a new metric to quantify this property, and develop the Sample-level-flatness-based Dataset Distillation (SFDD) approach, which filters the training data to retain only the most valuable samples. Experiments on the EAGLE framework demonstrate that SFDD can achieve over 2$\times$ training speedup using only 50% of the data, while keeping the final model's inference speedup within 4% of the full-dataset baseline. This work introduces an effective, data-centric approach that substantially improves the training efficiency for Speculative Decoding. Our code is available at https://anonymous.4open.science/r/Flatness.