Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics
This work provides a systematic framework for understanding and comparing fairness metrics in NLP, which is incremental but useful for researchers and practitioners aiming to quantify and address bias in AI systems.
The paper tackles the problem of measuring social biases in NLP models by unifying existing fairness metrics into three generalized forms and empirically comparing them, showing that differences in bias measurements can be explained by parameter choices in these generalized metrics.
Measuring bias is key for better understanding and addressing unfairness in NLP/ML models. This is often done via fairness metrics which quantify the differences in a model's behaviour across a range of demographic groups. In this work, we shed more light on the differences and similarities between the fairness metrics used in NLP. First, we unify a broad range of existing metrics under three generalized fairness metrics, revealing the connections between them. Next, we carry out an extensive empirical comparison of existing metrics and demonstrate that the observed differences in bias measurement can be systematically explained via differences in parameter choices for our generalized metrics.