A Source-Criticism Debiasing Method for GloVe Embeddings
This addresses bias in word embeddings for NLP applications, but it is incremental as it builds on existing debiasing methods with a novel approach.
The paper tackles the problem of social biases in GloVe word embeddings by introducing a debiasing method that incorporates explicit bias information instead of removing data, resulting in reduced bias effect sizes on WEAT tests without losing training data or TOP-1 performance.
It is well-documented that word embeddings trained on large public corpora consistently exhibit known human social biases. Although many methods for debiasing exist, almost all fixate on completely eliminating biased information from the embeddings and often diminish training set size in the process. In this paper, we present a simple yet effective method for debiasing GloVe word embeddings (Pennington et al., 2014) which works by incorporating explicit information about training set bias rather than removing biased data outright. Our method runs quickly and efficiently with the help of a fast bias gradient approximation method from Brunet et al. (2019). As our approach is akin to the notion of 'source criticism' in the humanities, we term our method Source-Critical GloVe (SC-GloVe). We show that SC-GloVe reduces the effect size on Word Embedding Association Test (WEAT) sets without sacrificing training data or TOP-1 performance.