Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space
For researchers in dataset distillation, this work provides a principled understanding of dataset effectiveness beyond distributional similarity, though the findings are incremental and domain-specific.
The paper investigates why some distilled datasets are more effective than others in dataset distillation, using discrete visual tokenizers to analyze token composition. It introduces a structural score that measures token composition adequacy and shows that balanced token composition correlates with higher validation performance, while divergence from original data does not necessarily harm performance.
Dataset distillation (DD) has proven to reduce training cost while preserving accuracy. While promising, the factors that make one distilled dataset more effective than another remain poorly understood. In this work, we investigate this question through the lens of discrete visual tokenizers. Whereas many prior DD efforts emphasize matching global data distributions, we suggest that the effectiveness depends on which semantic concepts are captured and how they are composed. Discrete visual tokenizers provide a finite vocabulary that enables direct statistical analysis of such compositional structure. Through quantitative analysis of token-level statistics, we introduce the structural score to measure the adequacy of token compositions. We observe that distilled datasets with balanced token composition yield higher validation performance. On the other hand, divergence from the original data does not necessarily harm performance. We further show that samples with high structural scores in the discrete token space can effectively guide diffusion-based DD. Our findings highlight the importance of token composition in dataset effectiveness, offering a principled complement to distributional similarity considerations in DD.