Bias Mitigation in Fine-tuning Pre-trained Models for Enhanced Fairness and Efficiency
This addresses fairness issues in fine-tuning for real-world applications, but it is incremental as it builds on existing transfer learning and bias mitigation techniques.
The paper tackles the problem of unfair outcomes when fine-tuning pre-trained models on new tasks by introducing a framework that mitigates biases through weight importance neutralization and matrix factorization, achieving effectiveness in experiments across multiple models and tasks.
Fine-tuning pre-trained models is a widely employed technique in numerous real-world applications. However, fine-tuning these models on new tasks can lead to unfair outcomes. This is due to the absence of generalization guarantees for fairness properties, regardless of whether the original pre-trained model was developed with fairness considerations. To tackle this issue, we introduce an efficient and robust fine-tuning framework specifically designed to mitigate biases in new tasks. Our empirical analysis shows that the parameters in the pre-trained model that affect predictions for different demographic groups are different, so based on this observation, we employ a transfer learning strategy that neutralizes the importance of these influential weights, determined using Fisher information across demographic groups. Additionally, we integrate this weight importance neutralization strategy with a matrix factorization technique, which provides a low-rank approximation of the weight matrix using fewer parameters, reducing the computational demands. Experiments on multiple pre-trained models and new tasks demonstrate the effectiveness of our method.