A Generic Framework for Conformal Fairness
This addresses fairness issues in machine learning for applications with non-IID data, such as graph data, but is incremental as it builds on existing conformal prediction methods.
The authors tackled the problem of ensuring fairness in uncertainty quantification by formalizing Conformal Fairness, a framework that controls coverage gaps between sensitive groups using conformal predictors, and demonstrated its effectiveness on graph and tabular datasets with theoretical alignment.
Conformal Prediction (CP) is a popular method for uncertainty quantification with machine learning models. While conformal prediction provides probabilistic guarantees regarding the coverage of the true label, these guarantees are agnostic to the presence of sensitive attributes within the dataset. In this work, we formalize \textit{Conformal Fairness}, a notion of fairness using conformal predictors, and provide a theoretically well-founded algorithm and associated framework to control for the gaps in coverage between different sensitive groups. Our framework leverages the exchangeability assumption (implicit to CP) rather than the typical IID assumption, allowing us to apply the notion of Conformal Fairness to data types and tasks that are not IID, such as graph data. Experiments were conducted on graph and tabular datasets to demonstrate that the algorithm can control fairness-related gaps in addition to coverage aligned with theoretical expectations.