Distilling Invariant Representations with Dual Augmentation
This work addresses the challenge of enhancing model efficiency and robustness in knowledge distillation for machine learning practitioners, though it is incremental as it builds on existing causal distillation methods.
The paper tackled the problem of improving knowledge distillation by promoting invariant feature learning through a dual augmentation strategy applied to both teacher and student models, achieving competitive results on CIFAR-100 in same-architecture distillation.
Knowledge distillation (KD) has been widely used to transfer knowledge from large, accurate models (teachers) to smaller, efficient ones (students). Recent methods have explored enforcing consistency by incorporating causal interpretations to distill invariant representations. In this work, we extend this line of research by introducing a dual augmentation strategy to promote invariant feature learning in both teacher and student models. Our approach leverages different augmentations applied to both models during distillation, pushing the student to capture robust, transferable features. This dual augmentation strategy complements invariant causal distillation by ensuring that the learned representations remain stable across a wider range of data variations and transformations. Extensive experiments on CIFAR-100 demonstrate the effectiveness of this approach, achieving competitive results in same-architecture KD.