Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias
This addresses exposure bias in knowledge distillation for machine learning practitioners, offering a novel approach to leverage teacher relational biases, though it is incremental in refining existing distillation methods.
The paper tackles the problem of knowledge distillation by proposing Group Relative Knowledge Distillation (GRKD), which focuses on learning relative class rankings from the teacher instead of absolute probabilities, leading to improved generalization in classification benchmarks, especially for fine-grained tasks.
Knowledge distillation typically transfers knowledge from a teacher model to a student model by minimizing differences between their output distributions. However, existing distillation approaches largely focus on mimicking absolute probabilities and neglect the valuable relational inductive biases embedded in the teacher's relative predictions, leading to exposure bias. In this paper, we propose Group Relative Knowledge Distillation (GRKD), a novel framework that distills teacher knowledge by learning the relative ranking among classes, rather than directly fitting the absolute distribution. Specifically, we introduce a group relative loss that encourages the student model to preserve the pairwise preference orderings provided by the teacher's outputs. Extensive experiments on classification benchmarks demonstrate that GRKD achieves superior generalization compared to existing methods, especially in tasks requiring fine-grained class differentiation. Our method provides a new perspective on exploiting teacher knowledge, focusing on relational structure rather than absolute likelihood.