Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets
For practitioners using dataset distillation with noisy labels, this method provides a robust, label-preserving approach that outperforms existing DD methods.
This paper tackles the problem of dataset distillation under noisy labels, where conventional methods amplify noise. The proposed trajectory-based framework with Selective Guidance Reweighting and Teacher-Inspired Auxiliary Targets achieves consistent gains over state-of-the-art DD baselines across symmetric, asymmetric, and real-world noise, improving robustness without clean subsets.
Dataset distillation (DD) condenses large corpora into compact, information-rich subsets for efficient training and reuse. However, under noisy supervision, DD risks condensing corrupted associations together with useful signals, degrading robustness. Conventional noisy-label remedies (sample selection, loss weighting, label correction) tightly couple noise estimation with model optimization, often require clean anchors, and can amplify confirmation bias-assumptions that are misaligned with DD's goal of compact, plug-and-play supervision. We therefore propose a trajectory-based DD framework that jointly suppresses noise and preserves transferable knowledge without relabeling or clean subsets. It comprises two complementary components: Selective Guidance Reweighting (SGR), which fuses global forgetting patterns (second-split forgetting) with local neighborhood consistency into a progressive reweighting scheme that prioritizes clean supervision along the teacher trajectory; and Teacher-Inspired Auxiliary Targets (TIAT), which inject auxiliary residual guidance distilled from intermediate teacher dynamics to reinforce informative signals while remaining internally consistent. Together, SGR and TIAT produce distilled datasets with cleaner and richer representations under noisy supervision. The framework is robust, label-preserving, computationally lightweight, and broadly applicable, yielding consistent gains over state-of-the-art DD baselines across symmetric, asymmetric, and real-world noise.