Assessing Disparate Impacts of Personalized Interventions: Identifiability and Bounds
This addresses fairness evaluation in high-stakes domains like social services and healthcare, offering a methodological advance for auditing personalized policies, though it is incremental in building on existing causal inference frameworks.
The paper tackles the problem of auditing disparate impacts of personalized interventions, showing that standard observational metrics are impossible to compute due to unknown ground truths, and provides identifiability results and bounds under monotone treatment response assumptions, demonstrated with a case study on a French job training dataset.
Personalized interventions in social services, education, and healthcare leverage individual-level causal effect predictions in order to give the best treatment to each individual or to prioritize program interventions for the individuals most likely to benefit. While the sensitivity of these domains compels us to evaluate the fairness of such policies, we show that actually auditing their disparate impacts per standard observational metrics, such as true positive rates, is impossible since ground truths are unknown. Whether our data is experimental or observational, an individual's actual outcome under an intervention different than that received can never be known, only predicted based on features. We prove how we can nonetheless point-identify these quantities under the additional assumption of monotone treatment response, which may be reasonable in many applications. We further provide a sensitivity analysis for this assumption by means of sharp partial-identification bounds under violations of monotonicity of varying strengths. We show how to use our results to audit personalized interventions using partially-identified ROC and xROC curves and demonstrate this in a case study of a French job training dataset.