A Possibility in Algorithmic Fairness: Can Calibration and Equal Error Rates Be Reconciled?
This work addresses fairness issues in high-stakes decision-making for marginalized groups, offering a practical solution to reconcile competing fairness criteria, though it is incremental in building on prior theoretical conflicts.
The paper tackles the conflict between calibration and error rate equality in algorithmic fairness for binary treatments like bail and loans, showing that calibrated scores can achieve equal error rates under certain conditions and presenting an algorithm that eliminates error disparities while maintaining calibration, as demonstrated with COMPAS and credit lending applications where it improved profit and loan access.
Decision makers increasingly rely on algorithmic risk scores to determine access to binary treatments including bail, loans, and medical interventions. In these settings, we reconcile two fairness criteria that were previously shown to be in conflict: calibration and error rate equality. In particular, we derive necessary and sufficient conditions for the existence of calibrated scores that yield classifications achieving equal error rates at any given group-blind threshold. We then present an algorithm that searches for the most accurate score subject to both calibration and minimal error rate disparity. Applied to the COMPAS criminal risk assessment tool, we show that our method can eliminate error disparities while maintaining calibration. In a separate application to credit lending, we compare our procedure to the omission of sensitive features and show that it raises both profit and the probability that creditworthy individuals receive loans.