A Bandit-Based Algorithm for Fairness-Aware Hyperparameter Optimization
This addresses the problem of real-world adoption of bias reduction methods for ML practitioners by providing an efficient tool for fairness-aware hyperparameter optimization.
The paper tackles the challenge of efficiently finding optimal fairness-accuracy trade-offs in machine learning by introducing Fairband, a bandit-based hyperparameter optimization algorithm. The result shows that Fairband consistently finds configurations with substantially improved fairness at a small decrease in accuracy on four real-world datasets, without extra training cost.
Considerable research effort has been guided towards algorithmic fairness but there is still no major breakthrough. In practice, an exhaustive search over all possible techniques and hyperparameters is needed to find optimal fairness-accuracy trade-offs. Hence, coupled with the lack of tools for ML practitioners, real-world adoption of bias reduction methods is still scarce. To address this, we present Fairband, a bandit-based fairness-aware hyperparameter optimization (HO) algorithm. Fairband is conceptually simple, resource-efficient, easy to implement, and agnostic to both the objective metrics, model types and the hyperparameter space being explored. Moreover, by introducing fairness notions into HO, we enable seamless and efficient integration of fairness objectives into real-world ML pipelines. We compare Fairband with popular HO methods on four real-world decision-making datasets. We show that Fairband can efficiently navigate the fairness-accuracy trade-off through hyperparameter optimization. Furthermore, without extra training cost, it consistently finds configurations attaining substantially improved fairness at a comparatively small decrease in predictive accuracy.