Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples
This work addresses the challenge of scalable and adaptable bias evaluation in NLP, though it is incremental as it builds on existing measures without introducing new ones.
The authors tackled the problem of comparing intrinsic gender bias evaluation measures for pre-trained language models without needing human-annotated examples, by creating bias-controlled models and computing rank correlations, and found that their method yields results comparable to those using human annotations.
Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators. To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples. Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using gender-related word lists. Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed. Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work.