Fair Performance Metric Elicitation
This addresses the challenge of defining fairness in machine learning for practitioners, though it is incremental as it builds on existing metric elicitation methods.
The paper tackles the problem of selecting fair performance metrics for multiclass classification with multiple sensitive groups by proposing a metric elicitation framework that allows practitioners to tune metrics based on task, context, and population, resulting in a strategy that requires only relative preference feedback and is robust to noise.
What is a fair performance metric? We consider the choice of fairness metrics through the lens of metric elicitation -- a principled framework for selecting performance metrics that best reflect implicit preferences. The use of metric elicitation enables a practitioner to tune the performance and fairness metrics to the task, context, and population at hand. Specifically, we propose a novel strategy to elicit group-fair performance metrics for multiclass classification problems with multiple sensitive groups that also includes selecting the trade-off between predictive performance and fairness violation. The proposed elicitation strategy requires only relative preference feedback and is robust to both finite sample and feedback noise.