FairBED: A Bayesian Experimental Design Approach to Gathering Fairer Data
For ML practitioners, FairBED addresses data-level bias by modifying data acquisition to inherently support fairer model training.
FairBED introduces a Bayesian experimental design approach that acquires data minimizing information about sensitive attributes while maximizing information about the target, leading to fairer datasets. Empirical results show improved fairness-accuracy trade-offs over random and conventional BED.
Frameworks for ensuring fairness in machine learning typically focus on learning fair models from existing data. But this endeavor is often undermined by biases already present in that data. We therefore look to modify the data acquisition process itself to help gather fairer data that is inherently more suitable for training fair predictors. To this end, we introduce FairBED, which provides novel formulations for quantifying the fairness of datasets themselves based on the idea that fair datasets should be uninformative about sensitive attributes. We then use this to construct practical fairness-aware Bayesian experimental design (BED) objectives that maximize expected information gain about the target quantity of interest while minimizing expected information gain about sensitive attributes. We further derive a theoretical link between FairBED and demographic parity, and show empirically that models trained on data gathered using FairBED provide improved fairness-accuracy trade-offs compared to randomly acquired data and conventional BED.