Evaluation of Social Biases in Recent Large Pre-Trained Models
This addresses harmful social biases in widely deployed AI models, but is incremental as it compares newer models to an existing baseline without introducing new methods.
The study evaluated social biases in recent large pre-trained models (ELECTRA, DeBERTa, DistilBERT) compared to BERT using StereoSet and CrowS-Pairs benchmarks, finding that while all models exhibit biases, they generally show improved bias reduction.
Large pre-trained language models are widely used in the community. These models are usually trained on unmoderated and unfiltered data from open sources like the Internet. Due to this, biases that we see in platforms online which are a reflection of those in society are in turn captured and learned by these models. These models are deployed in applications that affect millions of people and their inherent biases are harmful to the targeted social groups. In this work, we study the general trend in bias reduction as newer pre-trained models are released. Three recent models ( ELECTRA, DeBERTa, and DistilBERT) are chosen and evaluated against two bias benchmarks, StereoSet and CrowS-Pairs. They are compared to the baseline of BERT using the associated metrics. We explore whether as advancements are made and newer, faster, lighter models are released: are they being developed responsibly such that their inherent social biases have been reduced compared to their older counterparts? The results are compiled and we find that all the models under study do exhibit biases but have generally improved as compared to BERT.