An Empirical Study of Modular Bias Mitigators and Ensembles
This work addresses the problem of unstable fairness measurements in machine learning for practitioners, but it is incremental as it combines existing mitigators and ensembles without introducing new methods.
The paper tackled the instability of algorithmic bias mitigators across data splits by exploring their combination with ensemble methods, resulting in an open-source library for modular composition and empirical evaluation on 13 datasets, with distilled guidance for practitioners.
There are several bias mitigators that can reduce algorithmic bias in machine learning models but, unfortunately, the effect of mitigators on fairness is often not stable when measured across different data splits. A popular approach to train more stable models is ensemble learning. Ensembles, such as bagging, boosting, voting, or stacking, have been successful at making predictive performance more stable. One might therefore ask whether we can combine the advantages of bias mitigators and ensembles? To explore this question, we first need bias mitigators and ensembles to work together. We built an open-source library enabling the modular composition of 10 mitigators, 4 ensembles, and their corresponding hyperparameters. Based on this library, we empirically explored the space of combinations on 13 datasets, including datasets commonly used in fairness literature plus datasets newly curated by our library. Furthermore, we distilled the results into a guidance diagram for practitioners. We hope this paper will contribute towards improving stability in bias mitigation.