Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias
This work addresses fairness issues in machine learning systems for practitioners and researchers, highlighting the risks of feedback loops and proposing a mitigation strategy, though it is incremental in building on existing concepts like performative prediction.
The paper tackles the problem of model-induced distribution shifts (MIDS), where model outputs pollute training data over generations, leading to amplified bias and reduced fairness and performance, even in initially unbiased datasets. It introduces algorithmic reparation (AR) as a framework to mitigate these unfair feedback loops by curating representative training batches, demonstrating improvements in fairness.
Model-induced distribution shifts (MIDS) occur as previous model outputs pollute new model training sets over generations of models. This is known as model collapse in the case of generative models, and performative prediction or unfairness feedback loops for supervised models. When a model induces a distribution shift, it also encodes its mistakes, biases, and unfairnesses into the ground truth of its data ecosystem. We introduce a framework that allows us to track multiple MIDS over many generations, finding that they can lead to loss in performance, fairness, and minoritized group representation, even in initially unbiased datasets. Despite these negative consequences, we identify how models might be used for positive, intentional, interventions in their data ecosystems, providing redress for historical discrimination through a framework called algorithmic reparation (AR). We simulate AR interventions by curating representative training batches for stochastic gradient descent to demonstrate how AR can improve upon the unfairnesses of models and data ecosystems subject to other MIDS. Our work takes an important step towards identifying, mitigating, and taking accountability for the unfair feedback loops enabled by the idea that ML systems are inherently neutral and objective.