Fairness Improvement with Multiple Protected Attributes: How Far Are We?
This highlights a critical gap in fairness research for users with multiple protected attributes, showing current methods are inadequate and incremental in addressing real-world complexity.
The paper investigates fairness improvement in machine learning for multiple protected attributes, finding that optimizing for a single attribute often reduces fairness for others (up to 88.3% of scenarios) while accuracy loss remains similar, but F1-score impact doubles with two attributes.
Existing research mostly improves the fairness of Machine Learning (ML) software regarding a single protected attribute at a time, but this is unrealistic given that many users have multiple protected attributes. This paper conducts an extensive study of fairness improvement regarding multiple protected attributes, covering 11 state-of-the-art fairness improvement methods. We analyze the effectiveness of these methods with different datasets, metrics, and ML models when considering multiple protected attributes. The results reveal that improving fairness for a single protected attribute can largely decrease fairness regarding unconsidered protected attributes. This decrease is observed in up to 88.3% of scenarios (57.5% on average). More surprisingly, we find little difference in accuracy loss when considering single and multiple protected attributes, indicating that accuracy can be maintained in the multiple-attribute paradigm. However, the effect on F1-score when handling two protected attributes is about twice that of a single attribute. This has important implications for future fairness research: reporting only accuracy as the ML performance metric, which is currently common in the literature, is inadequate.