Using Early Readouts to Mediate Featural Bias in Distillation
This addresses fairness and accuracy issues in distillation for machine learning practitioners, but it is incremental as it builds on existing methods for mitigating bias.
The paper tackled the problem of spurious feature-label correlations in distillation, which worsen when students have less capacity than teachers, and showed that using early readouts to modulate loss improves group fairness and overall accuracy on benchmark datasets.
Deep networks tend to learn spurious feature-label correlations in real-world supervised learning tasks. This vulnerability is aggravated in distillation, where a student model may have lesser representational capacity than the corresponding teacher model. Often, knowledge of specific spurious correlations is used to reweight instances & rebalance the learning process. We propose a novel early readout mechanism whereby we attempt to predict the label using representations from earlier network layers. We show that these early readouts automatically identify problem instances or groups in the form of confident, incorrect predictions. Leveraging these signals to modulate the distillation loss on an instance level allows us to substantially improve not only group fairness measures across benchmark datasets, but also overall accuracy of the student model. We also provide secondary analyses that bring insight into the role of feature learning in supervision and distillation.