Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements
This work addresses the gap between theoretical fairness research and practical implementation for machine learning practitioners and affected users, though it is incremental in nature.
The paper tackles the challenge of applying algorithmic fairness principles to a real-world production classification system, introducing a new fairness metric called conditional equality and a training approach that improves fairness performance.
As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of the concerns and challenges in deploying machine learning, but there has been much less work in seeing how the rubber meets the road. In this paper we provide a case-study on the application of fairness in machine learning research to a production classification system, and offer new insights in how to measure and address algorithmic fairness issues. We discuss open questions in implementing equality of opportunity and describe our fairness metric, conditional equality, that takes into account distributional differences. Further, we provide a new approach to improve on the fairness metric during model training and demonstrate its efficacy in improving performance for a real-world product