Fair When Trained, Unfair When Deployed: Observable Fairness Measures are Unstable in Performative Prediction Settings
This addresses a critical issue for practitioners in fields like criminal justice and healthcare, where fairness is essential but current measures may fail in performative settings, though it is incremental in proposing counterfactual-based solutions.
The paper tackles the problem of algorithmic fairness measures becoming unstable under distribution shifts induced by predictor deployment, showing that predictors fair at training can become unfair when deployed due to concept shift, with theoretical and simulated evidence.
Many popular algorithmic fairness measures depend on the joint distribution of predictions, outcomes, and a sensitive feature like race or gender. These measures are sensitive to distribution shift: a predictor which is trained to satisfy one of these fairness definitions may become unfair if the distribution changes. In performative prediction settings, however, predictors are precisely intended to induce distribution shift. For example, in many applications in criminal justice, healthcare, and consumer finance, the purpose of building a predictor is to reduce the rate of adverse outcomes such as recidivism, hospitalization, or default on a loan. We formalize the effect of such predictors as a type of concept shift-a particular variety of distribution shift-and show both theoretically and via simulated examples how this causes predictors which are fair when they are trained to become unfair when they are deployed. We further show how many of these issues can be avoided by using fairness definitions that depend on counterfactual rather than observable outcomes.