ABROCA Distributions For Algorithmic Bias Assessment: Considerations Around Interpretation
This work addresses the problem of misinterpretation in fairness metrics for researchers and practitioners in learning analytics, highlighting incremental considerations for using ABROCA.
The study analyzed the statistical properties of the ABROCA metric for algorithmic bias assessment, finding that its distributions are highly skewed due to factors like sample sizes and class imbalance, which can inflate bias estimates by chance even when no true bias exists.
Algorithmic bias continues to be a key concern of learning analytics. We study the statistical properties of the Absolute Between-ROC Area (ABROCA) metric. This fairness measure quantifies group-level differences in classifier performance through the absolute difference in ROC curves. ABROCA is particularly useful for detecting nuanced performance differences even when overall Area Under the ROC Curve (AUC) values are similar. We sample ABROCA under various conditions, including varying AUC differences and class distributions. We find that ABROCA distributions exhibit high skewness dependent on sample sizes, AUC differences, and class imbalance. When assessing whether a classifier is biased, this skewness inflates ABROCA values by chance, even when data is drawn (by simulation) from populations with equivalent ROC curves. These findings suggest that ABROCA requires careful interpretation given its distributional properties, especially when used to assess the degree of bias and when classes are imbalanced.