Evaluating the fairness of fine-tuning strategies in self-supervised learning
This addresses fairness issues in fine-tuning for practitioners using SSL models, though it is incremental as it builds on existing supervised learning techniques.
The paper tackles the problem of fairness degradation during fine-tuning of self-supervised learning models, finding that updating only Batch Normalization statistics improves downstream fairness by 36% in worst subgroup and 25% in mean subgroup gap, while being competitive with supervised learning and requiring far fewer parameters and training time.
In this work we examine how fine-tuning impacts the fairness of contrastive Self-Supervised Learning (SSL) models. Our findings indicate that Batch Normalization (BN) statistics play a crucial role, and that updating only the BN statistics of a pre-trained SSL backbone improves its downstream fairness (36% worst subgroup, 25% mean subgroup gap). This procedure is competitive with supervised learning, while taking 4.4x less time to train and requiring only 0.35% as many parameters to be updated. Finally, inspired by recent work in supervised learning, we find that updating BN statistics and training residual skip connections (12.3% of the parameters) achieves parity with a fully fine-tuned model, while taking 1.33x less time to train.