Invariance and identifiability issues for word embeddings
This addresses a foundational problem in natural language processing for researchers and practitioners by highlighting that performance comparisons may be misleading due to inherent invariance mismatches.
The paper tackles the problem of performance disparities in word embeddings by identifying that they are defined only up to transformations that leave the training criterion invariant, which may not align with evaluation functions, leading to arbitrary superiority claims. It provides a formal analysis and numerical examples to illustrate this identifiability issue.
Word embeddings are commonly obtained as optimizers of a criterion function $f$ of a text corpus, but assessed on word-task performance using a different evaluation function $g$ of the test data. We contend that a possible source of disparity in performance on tasks is the incompatibility between classes of transformations that leave $f$ and $g$ invariant. In particular, word embeddings defined by $f$ are not unique; they are defined only up to a class of transformations to which $f$ is invariant, and this class is larger than the class to which $g$ is invariant. One implication of this is that the apparent superiority of one word embedding over another, as measured by word task performance, may largely be a consequence of the arbitrary elements selected from the respective solution sets. We provide a formal treatment of the above identifiability issue, present some numerical examples, and discuss possible resolutions.