Quadratic Metric Elicitation for Fairness and Beyond
This work addresses the need for more flexible performance metrics in machine learning, especially for fairness, by enabling elicitation of quadratic functions, though it is incremental as it builds on existing metric elicitation frameworks.
The paper tackles the limitation of linear metric elicitation by developing a strategy for eliciting quadratic and polynomial metrics, which better reflect human preferences, particularly for fairness applications, achieving near-optimal query complexity with robustness to noise.
Metric elicitation is a recent framework for eliciting classification performance metrics that best reflect implicit user preferences based on the task and context. However, available elicitation strategies have been limited to linear (or quasi-linear) functions of predictive rates, which can be practically restrictive for many applications including fairness. This paper develops a strategy for eliciting more flexible multiclass metrics defined by quadratic functions of rates, designed to reflect human preferences better. We show its application in eliciting quadratic violation-based group-fair metrics. Our strategy requires only relative preference feedback, is robust to noise, and achieves near-optimal query complexity. We further extend this strategy to eliciting polynomial metrics -- thus broadening the use cases for metric elicitation.