Effect of dimensionality change on the bias of word embeddings
This addresses bias issues in word embeddings for NLP practitioners, but it is incremental as it builds on existing dimensionality research.
The study investigated how changing the dimensionality of word embeddings affects their bias, using static and context-sensitive models on the English Wikipedia corpus, finding significant variation and no uniformity in bias changes with dimensionality.
Word embedding methods (WEMs) are extensively used for representing text data. The dimensionality of these embeddings varies across various tasks and implementations. The effect of dimensionality change on the accuracy of the downstream task is a well-explored question. However, how the dimensionality change affects the bias of word embeddings needs to be investigated. Using the English Wikipedia corpus, we study this effect for two static (Word2Vec and fastText) and two context-sensitive (ElMo and BERT) WEMs. We have two observations. First, there is a significant variation in the bias of word embeddings with the dimensionality change. Second, there is no uniformity in how the dimensionality change affects the bias of word embeddings. These factors should be considered while selecting the dimensionality of word embeddings.